Psychometric Tests: Quantifying the Consequences of Low Reliability and Improving Reliability Estimation
Abstract
We introduce a set of consequence-based measures that quantify the misclassification arising from imperfect reliability in multi-item psychometric scales. Particular attention is given to the probability of individuals in extreme latent-trait percentiles being correctly identified from their observed scores. These cost functions provide a principled and interpretable way to characterise how inadequate reliability distorts classification and reduces the informational value of test scores. We also examine methodological issues in estimating reliability under common-factor models, with emphasis on McDonald's omega and the construction of accurate confidence intervals. To address the computational burden of model fitting, especially in small samples, we derive a modified version of Cronbach's alpha that closely approximates omega when the common-factor model holds. This estimator has comparable sampling variability to omega while requiring no parameter estimation, offering a practical and computationally efficient alternative. y
Related
- Metatheoretical multiverse analysis: Improving the reliability of research with theories-as-data
- EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
- High-dimensional reliability-oriented Shapley effects estimation
- Reliability Modeling using an Equilibrium-Based Xgamma Distribution: Properties, Estimation, and Applications
- The LR2 Lifetime Distribution: Rate Crossing, Reliability Properties, and Estimation
BibTeX
@misc{baker2026psychometric,
title = {{Psychometric Tests: Quantifying the Consequences of Low Reliability and Improving Reliability Estimation}},
author = {Rose Baker},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.03006},
eprint = {2610.03006},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.03006},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Psychometric Tests: Quantifying the Consequences of Low Reliability and Improving Reliability Estimation
AU - Rose Baker
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2610.03006
UR - https://arxiv.org/abs/2610.03006
AB - We introduce a set of consequence-based measures that quantify the misclassification arising from imperfect reliability in multi-item psychometric scales. Particular attention is given to the probability of individuals in extreme latent-trait percentiles being correctly identified from their observed scores. These cost functions provide a principled and interpretable way to characterise how inadequate reliability distorts classification and reduces the informational value of test scores. We also examine methodological issues in estimating reliability under common-factor models, with emphasis on McDonald's omega and the construction of accurate confidence intervals. To address the computational burden of model fitting, especially in small samples, we derive a modified version of Cronbach's alpha that closely approximates omega when the common-factor model holds. This estimator has comparable sampling variability to omega while requiring no parameter estimation, offering a practical and computationally efficient alternative. y
ER -