Measurement reliability, construct validity, and transparent reporting in original and replication psychological research
- 1Department of Methodology and Statistics, School of Social and Behavioral Sciences, Tilburg University, Tilburg, Netherlands
Abstract
Published (replication) studies that use measures to assess psychological phenomena need to transparently report information on measurement procedures, validity, and reliability to allow verification and use in future (replication) research. However, earlier results highlighted widespread poor reporting of psychological measurement. Here, we investigated measurement reporting in a sample of 77 measures within 56 Many Labs replications and related original articles (14–17) and found that the information relevant for reusing measures was reported in full in around half the replication measures, and only in 5.2% of the original studies. We also observed that around a third of multiple-item measures in original studies and 11.4% in replications reported reliability coefficients, with comparable proportions for reporting any convergent, discriminant, predictive, or factorial validity evidence. We assessed the reliability, unidimensionality, and measurement invariance of multiple-item measures using the openly available Many Labs item response data. We observed that while some measures passed these psychometric checks, they rarely did so consistently across labs. These results corroborate existing findings that measurement reporting in published research lacks transparency, and that poor measurement reporting may obscure insufficient reliability and validity. We offer suggestions on how to improve measurement reporting practices and increase the use of validated measures.
Related
- Reproducibility is not construct validity: LLM measurement of institutionally situated communication
- A general method for estimating reliability using Bayesian Measurement Uncertainty
- Recovering Hierarchical Structures in Psychological Measurement
- On the use of the Belopol'skaya-Daletskii representation of a diffusion on a Riemann manifold to construct path integrals
- Using the slowest observable in one-dimensional Markov processes to construct quasi-exactly-solvable generators with two explicit levels
BibTeX
@article{goos2026measurement,
title = {{Measurement reliability, construct validity, and transparent reporting in original and replication psychological research}},
author = {Cas Goos and Marjan Bakker and Jelte M. Wicherts and Michèle B. Nuijten},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0356070},
url = {https://doi.org/10.1371/journal.pone.0356070},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Measurement reliability, construct validity, and transparent reporting in original and replication psychological research
AU - Cas Goos
AU - Marjan Bakker
AU - Jelte M. Wicherts
AU - Michèle B. Nuijten
PY - 2026
DA - 2026/09/02
JO - PLOS One
DO - 10.1371/journal.pone.0356070
UR - https://doi.org/10.1371/journal.pone.0356070
AB - Published (replication) studies that use measures to assess psychological phenomena need to transparently report information on measurement procedures, validity, and reliability to allow verification and use in future (replication) research. However, earlier results highlighted widespread poor reporting of psychological measurement. Here, we investigated measurement reporting in a sample of 77 measures within 56 Many Labs replications and related original articles (14–17) and found that the information relevant for reusing measures was reported in full in around half the replication measures, and only in 5.2% of the original studies. We also observed that around a third of multiple-item measures in original studies and 11.4% in replications reported reliability coefficients, with comparable proportions for reporting any convergent, discriminant, predictive, or factorial validity evidence. We assessed the reliability, unidimensionality, and measurement invariance of multiple-item measures using the openly available Many Labs item response data. We observed that while some measures passed these psychometric checks, they rarely did so consistently across labs. These results corroborate existing findings that measurement reporting in published research lacks transparency, and that poor measurement reporting may obscure insufficient reliability and validity. We offer suggestions on how to improve measurement reporting practices and increase the use of validated measures.
SN - 1932-6203
ER -