Fixed Raters, Random Selection: A Correlation-Based Framework for Agreement ICCs in Two-Way Mixed-Effects Designs
Abstract
Intraclass correlation coefficients (ICCs) quantify the reliability of measurements made on the same persons by different raters. In interrater reliability studies, consistency ICCs assess whether scores preserve the relative rankings of persons, while agreement ICCs evaluate the closeness of absolute scores across raters, thereby penalising systematic rater differences. In two-way mixed-effects designs, where raters are treated as fixed, conventional agreement ICCs raise a conceptual paradox: fixed-rater dispersion contributes to a source of disagreement despite the rater effects being constants, and it remains in the coefficient for all-rater averages even though those effects cancel in the average. In this paper, we resolve this paradox by introducing a finite-population sampling framework grounded in the correlation approach with hypothetical replications. Within this framework, we derive correlation-based agreement ICCs for both single ratings and all-rater averages, establishing a coherent probabilistic foundation that resolves the paradox. By considering various rater-sampling schemes, a family of agreement ICCs in mixed-effects designs is presented. We further show that the conventional agreement formulas can be recovered under independent sampling with replacement both within and between replicates. Thus, choosing an appropriate agreement ICC requires specifying how raters are sampled within and across replicates, rather than relying on the design alone.
Related
- A weighted casebase framework for outcome-dependent two-phase sampling designs.
- Qualify-Then-Borrow: A Five-Step Framework for Bayesian Borrowing Beyond Outcome Agreement
- On the Pricing of American Options under Stochastic Local Volatility and Stochastic Correlation via the RBSDE Framework
- ACS: An interactive framework for conformal selection
- From Corridor Selection to Earthwork: A Multi-Stage Framework for Automated Road Design via Steiner Trees and Convex Optimization
BibTeX
@misc{lin2026fixed,
title = {{Fixed Raters, Random Selection: A Correlation-Based Framework for Agreement ICCs in Two-Way Mixed-Effects Designs}},
author = {Tzu-Yao Lin and francis tuerlinckx and Sophie Vanbelle},
howpublished = {PsyArXiv},
year = {2026},
month = nov,
doi = {10.31234/osf.io/2ctua_v1},
url = {https://osf.io/preprints/psyarxiv/2ctua_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Fixed Raters, Random Selection: A Correlation-Based Framework for Agreement ICCs in Two-Way Mixed-Effects Designs
AU - Tzu-Yao Lin
AU - francis tuerlinckx
AU - Sophie Vanbelle
PY - 2026
DA - 2026/11/05
PB - PsyArXiv
DO - 10.31234/osf.io/2ctua_v1
UR - https://osf.io/preprints/psyarxiv/2ctua_v1/
AB - Intraclass correlation coefficients (ICCs) quantify the reliability of measurements made on the same persons by different raters. In interrater reliability studies, consistency ICCs assess whether scores preserve the relative rankings of persons, while agreement ICCs evaluate the closeness of absolute scores across raters, thereby penalising systematic rater differences. In two-way mixed-effects designs, where raters are treated as fixed, conventional agreement ICCs raise a conceptual paradox: fixed-rater dispersion contributes to a source of disagreement despite the rater effects being constants, and it remains in the coefficient for all-rater averages even though those effects cancel in the average. In this paper, we resolve this paradox by introducing a finite-population sampling framework grounded in the correlation approach with hypothetical replications. Within this framework, we derive correlation-based agreement ICCs for both single ratings and all-rater averages, establishing a coherent probabilistic foundation that resolves the paradox. By considering various rater-sampling schemes, a family of agreement ICCs in mixed-effects designs is presented. We further show that the conventional agreement formulas can be recovered under independent sampling with replacement both within and between replicates. Thus, choosing an appropriate agreement ICC requires specifying how raters are sampled within and across replicates, rather than relying on the design alone.
ER -