How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
Abstract
LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statistical evidence of a provider's advantage over a fixed comparator. For a fixed candidate family under a Gaussian margin model, we derive a sensitivity curve that reports this maximum count as a function of a lower bound on within-family correlation. The relevant correlation must match the score used for ranking and the sampling model: in a controlled family, pooled item correlation is 0.90, whereas composite-score correlation is 0.46 under item resampling and 0.92 when MMLU subjects are resampled. An item-based audit of 394 adjacent-rank claims on the Open LLM Leaderboard finds that 391 lack statistical support even before accounting for selection. Among claims that pass the uncorrected test, certification can depend on assumptions about the hidden family's correlation. The resulting curves make these assumptions explicit without estimating the unobserved search size.
Related
- Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging
- Cost-Sensitive Online Window Size Selection for Portfolio Management
- Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
- Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel
- Asymptotics for Bivariate Risk Models with Random-Size Clusters of Dependent Claims
BibTeX
@misc{yang2026sensitive,
title = {{How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?}},
author = {Chen Yang and Jun Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28177},
eprint = {2609.28177},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28177},
}
RIS
TY - UNPB
TI - How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
AU - Chen Yang
AU - Jun Chen
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.28177
UR - https://arxiv.org/abs/2609.28177
AB - LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statistical evidence of a provider's advantage over a fixed comparator. For a fixed candidate family under a Gaussian margin model, we derive a sensitivity curve that reports this maximum count as a function of a lower bound on within-family correlation. The relevant correlation must match the score used for ranking and the sampling model: in a controlled family, pooled item correlation is 0.90, whereas composite-score correlation is 0.46 under item resampling and 0.92 when MMLU subjects are resampled. An item-based audit of 394 adjacent-rank claims on the Open LLM Leaderboard finds that 391 lack statistical support even before accounting for selection. Among claims that pass the uncorrected test, certification can depend on assumptions about the hidden family's correlation. The resulting curves make these assumptions explicit without estimating the unobserved search size.
ER -