Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation
Abstract
A rigorous statistical treatment of two-model comparisons on the same evals can be achieved by paired t-tests, analyzing their standard errors and a clustering correction for correlated questions. Nevertheless, leaderboards, ablations studies, and hyperparameter sweeps, usually compare K>2 models simultaneously. In this paper, we present a single random-effects model for scores on shared evals. We leverage models as a fixed effects and questions (or question cluster) as random effects. We show that fitting it a classical ANOVA or a linear mixed model we can recover Miller's paired and clustered estimators for K=2, but we extend the results to any K and to unbalanced, clustered designs. We validate the described model in a simulation study and using a real-data application. We take six openly available language models scored on 1,497 shared MMLU-Pro questions across 14 subject clusters, and we show that pairwise ranking claims survive depending on properly accounting for both question-level pairing and multiple comparisons. By the end of the manuscript, we provide concrete recommendations for reporting multi-model eval results.
Related
- A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
- Comparison of statistical methods for high-dimensional compositional data from flow cytometry: A critical perspective on log-ratio transformation and differential abundance testing
- Statistical comparison of reconstruction methods for the inverse boundary problem of the one-dimensional wave equation
- Riemannian Newton Methods with Statistical Inference
- Statistical Sheaf Modeling Methods for Measuring Qualia Structures
BibTeX
@misc{reyes2026statistical,
title = {{Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation}},
author = {Juan Francisco Mandujano Reyes},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.22659},
eprint = {2608.22659},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.22659},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation
AU - Juan Francisco Mandujano Reyes
PY - 2026
DA - 2026/10/18
PB - arXiv
DO - 10.48550/arxiv.2608.22659
UR - https://arxiv.org/abs/2608.22659
AB - A rigorous statistical treatment of two-model comparisons on the same evals can be achieved by paired t-tests, analyzing their standard errors and a clustering correction for correlated questions. Nevertheless, leaderboards, ablations studies, and hyperparameter sweeps, usually compare K>2 models simultaneously. In this paper, we present a single random-effects model for scores on shared evals. We leverage models as a fixed effects and questions (or question cluster) as random effects. We show that fitting it a classical ANOVA or a linear mixed model we can recover Miller's paired and clustered estimators for K=2, but we extend the results to any K and to unbalanced, clustered designs. We validate the described model in a simulation study and using a real-data application. We take six openly available language models scored on 1,497 shared MMLU-Pro questions across 14 subject clusters, and we show that pairwise ranking claims survive depending on properly accounting for both question-level pairing and multiple comparisons. By the end of the manuscript, we provide concrete recommendations for reporting multi-model eval results.
ER -