A general method for estimating reliability using Bayesian Measurement Uncertainty
Abstract
Unreliable measurement can lead to lower statistical power, attenuated effect sizes and residual confounding. However, estimating reliability can be challenging for complex cognitive and behavioural assessments without test-retest data. Most statistical methods for assessing reliability with a single test administration are designed for fixed-item questionnaires. We introduce a novel Bayesian procedure called relative measurement uncertainty (RMU) for estimating reliability that can be broadly applied, including to many widely used computational models. Our approach draws pairs of samples from each subject's posterior and calculates the correlation between draws. We demonstrate analytically and via simulation that this method provides accurate reliability estimates and well-calibrated credible intervals across linear factor, signal detection and reinforcement learning models. Simulations found that RMU was more accurate (lower root mean square error) than existing measures (coefficient alpha, coefficient H, split-half), but more importantly, it can be applied across a wide range of computational models. RMU offers a general method for estimating reliability, leveraging the modelling flexibility of Bayesian statistical methods.
Related
- OmegaSwitch: Bayesian Markov-Modulated Codon Models for Estimating dN/dS
- Small Average Error, Wide Individual Error: General-Purpose LLMs Estimating Energy Intake from Meal Photos in a Feeding Trial
- Embracing uncertainty in time: A Bayesian dynamic mixing model for isotopic diet estimation
- Measurement reliability bounds functional benchmarks and relocates where variant effect prediction fails
- Uncertainty-Aware Model Selection with a Calibrated Probability-Generating-Function-Based Bayesian Information Criterion
BibTeX
@misc{bignardi2026general,
title = {{A general method for estimating reliability using Bayesian Measurement Uncertainty}},
author = {Giacomo Bignardi and Rogier Kievit and Paul-Christian Bürkner},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/h54k8_v2},
url = {https://osf.io/preprints/psyarxiv/h54k8_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A general method for estimating reliability using Bayesian Measurement Uncertainty
AU - Giacomo Bignardi
AU - Rogier Kievit
AU - Paul-Christian Bürkner
PY - 2026
DA - 2026/10/14
PB - PsyArXiv
DO - 10.31234/osf.io/h54k8_v2
UR - https://osf.io/preprints/psyarxiv/h54k8_v2/
AB - Unreliable measurement can lead to lower statistical power, attenuated effect sizes and residual confounding. However, estimating reliability can be challenging for complex cognitive and behavioural assessments without test-retest data. Most statistical methods for assessing reliability with a single test administration are designed for fixed-item questionnaires. We introduce a novel Bayesian procedure called relative measurement uncertainty (RMU) for estimating reliability that can be broadly applied, including to many widely used computational models. Our approach draws pairs of samples from each subject's posterior and calculates the correlation between draws. We demonstrate analytically and via simulation that this method provides accurate reliability estimates and well-calibrated credible intervals across linear factor, signal detection and reinforcement learning models. Simulations found that RMU was more accurate (lower root mean square error) than existing measures (coefficient alpha, coefficient H, split-half), but more importantly, it can be applied across a wide range of computational models. RMU offers a general method for estimating reliability, leveraging the modelling flexibility of Bayesian statistical methods.
ER -