Valid and Reliable? Basic and Expanded Recommendations for Psychometric Reporting and Quality Assessment
Abstract
Improving the validity, reliability, and replicability of research in the social and health sciences depends on the quality of measurement. We argue that judging whether the sum score of a multi-item measure adequately represents the intended latent variable rests on four basic criteria — unidimensionality, local independence of items, ordered response categories, and invariance — which can serve as a minimal checklist for end users assessing the quality of a psychometric study. A central recommendation is to stop interpreting model and item fit against rule-of-thumb critical values and instead use simulation-based cutoffs tailored to the data at hand. Because most published psychometric work relies on rule-of-thumb cutoffs, established scales are in urgent need of re-analysis with appropriate methods. We provide an up-to-date overview of methods for evaluating each criterion within Rasch Measurement Theory, point to freely available software that makes these methods accessible, and note outdated approaches to avoid. A draft reporting guideline is provided, as we believe that a consensus-based reporting standard may facilitate more transparent evaluations of scales. Finally, we argue for a higher bar for calling a measure "valid and reliable": the existence of an original validation paper, often old and based on outdated methods, is not in itself sufficient evidence of validity.
Related
- Multicenter Assessment of Combined Brain and Cervical Spinal cord 3T MP2RAGE T1 Measurements for Reliable Tissue Microstructure Quantification
- Rapid health assessment of coral reefs: a standardised, quick, and reliable tool developed and tested during the 2024/25 global bleaching event
- Psychometric assessment of the Persian version of the DUSAETCS questionnaire: Duke University Self-Assessment Educational Technology Competencies
- Limitations in the Assessment of Pain Reporting Reliability-Validation of the Focused Analgesia Selection Test.
- Cross-cultural adaptation and psychometric validation of the Kannada version of the Extended Effort Assessment Scale (EEAS-K) in adults with hearing loss
BibTeX
@misc{johansson2026valid,
title = {{Valid and Reliable? Basic and Expanded Recommendations for Psychometric Reporting and Quality Assessment}},
author = {Magnus Johansson and Marit Preuter and Jeanette Melin},
howpublished = {OSF Preprints},
year = {2026},
month = sep,
doi = {10.31219/osf.io/3htzc_v2},
url = {https://osf.io/3htzc_v2/},
note = {CC0},
}
RIS
TY - UNPB
TI - Valid and Reliable? Basic and Expanded Recommendations for Psychometric Reporting and Quality Assessment
AU - Magnus Johansson
AU - Marit Preuter
AU - Jeanette Melin
PY - 2026
DA - 2026/09/03
PB - OSF Preprints
DO - 10.31219/osf.io/3htzc_v2
UR - https://osf.io/3htzc_v2/
AB - Improving the validity, reliability, and replicability of research in the social and health sciences depends on the quality of measurement. We argue that judging whether the sum score of a multi-item measure adequately represents the intended latent variable rests on four basic criteria — unidimensionality, local independence of items, ordered response categories, and invariance — which can serve as a minimal checklist for end users assessing the quality of a psychometric study. A central recommendation is to stop interpreting model and item fit against rule-of-thumb critical values and instead use simulation-based cutoffs tailored to the data at hand. Because most published psychometric work relies on rule-of-thumb cutoffs, established scales are in urgent need of re-analysis with appropriate methods. We provide an up-to-date overview of methods for evaluating each criterion within Rasch Measurement Theory, point to freely available software that makes these methods accessible, and note outdated approaches to avoid. A draft reporting guideline is provided, as we believe that a consensus-based reporting standard may facilitate more transparent evaluations of scales. Finally, we argue for a higher bar for calling a measure "valid and reliable": the existence of an original validation paper, often old and based on outdated methods, is not in itself sufficient evidence of validity.
ER -