What Generalizes in Reliability Generalization?
Abstract
Zhao and Plonsky's synthesis of task-engagement research uses a distribution of reported reliability coefficients to assess measurement quality and discuss minimum acceptability. Such judgments require more than a position in that distribution. As the authors note, the pooled coefficients don't all summarize the same measurement property, and even comparable coefficients have different consequences for different scientific targets. Using the synthesis as a worked case, this response connects the authors' cautions about coefficient choice to their rationale for a percentile benchmark. Under explicit classical-error models, reliability of .6 leaves a randomized raw mean contrast unbiased but reduces its effective sample size to 60% of the original. An uncorrected slope on a noisy predictor instead converges to 60% of the latent slope. Standardized differences and correlations require further distinctions. These familiar results show why a field percentile alone can't tell a researcher what a planned study can learn. A worked planning example shows how, under the stated model, reliability and sample size together set the precision of an experimental contrast. Reliability syntheses can better support such decisions by displaying comparable coefficients separately and connecting them to uncertainty about specified estimates and predictions.
Related
- Uniform generalization of Goodstein's theorem and Cichon's independence proof
- On a generalization of the Reidemeister spectrum of a finite group
- On Reversible Rings, Their Generalization and Quasi-Idempotent Elements
- A Generalization of Amari's Bayesian Duality
- A Diophantine generalization of Fourier's irrationality trick
BibTeX
@misc{reynolds2026what,
title = {{What Generalizes in Reliability Generalization?}},
author = {Brett Reynolds},
howpublished = {PsyArXiv},
year = {2026},
month = nov,
doi = {10.31234/osf.io/zw4tq_v1},
url = {https://osf.io/preprints/psyarxiv/zw4tq_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - What Generalizes in Reliability Generalization?
AU - Brett Reynolds
PY - 2026
DA - 2026/11/02
PB - PsyArXiv
DO - 10.31234/osf.io/zw4tq_v1
UR - https://osf.io/preprints/psyarxiv/zw4tq_v1/
AB - Zhao and Plonsky's synthesis of task-engagement research uses a distribution of reported reliability coefficients to assess measurement quality and discuss minimum acceptability. Such judgments require more than a position in that distribution. As the authors note, the pooled coefficients don't all summarize the same measurement property, and even comparable coefficients have different consequences for different scientific targets. Using the synthesis as a worked case, this response connects the authors' cautions about coefficient choice to their rationale for a percentile benchmark. Under explicit classical-error models, reliability of .6 leaves a randomized raw mean contrast unbiased but reduces its effective sample size to 60% of the original. An uncorrected slope on a noisy predictor instead converges to 60% of the latent slope. Standardized differences and correlations require further distinctions. These familiar results show why a field percentile alone can't tell a researcher what a planned study can learn. A worked planning example shows how, under the stated model, reliability and sample size together set the precision of an experimental contrast. Reliability syntheses can better support such decisions by displaying comparable coefficients separately and connecting them to uncertainty about specified estimates and predictions.
ER -