Accounting for Bias Enables Sustainable LLM Evaluation

Abstract

LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

D3MI: an efficient and powerful federated imputation method for bias reduction in the analysis of distributed incomplete data by accounting for within-site correlation and between-site heterogeneity.
Yi Lian, Xiaoqian Jiang, Qi Long
BMC medical informatics and decision making8 Oct 2026Peer-reviewedCC BYFull textStatisticsEurope PMC
Difference-in-Differences for Heterogeneous Effects of General Treatment Strategies Accounting for Time-Varying Covariates
Oliver Dukes, Zach Shahn, James Robins +1
arXiv15 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Tight End-to-End Accounting for Differentially Private Synthetic Data
Georgi Ganev, Vadym Doroshenko, Arun Ganesh +3
arXiv12 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Exact information accounting for SGD methods
Akshay Balsubramani
arXiv1 Oct 2026PreprintarXiv licenceFull textOptimisation & control StatisticsarXiv
Problems of Automated Control of Viral Disease Progression Accounting for Spatial Perturbations and Body Temperature Response Under Immunotherapy
Serhii Baranovsky, Andrii Bomba, N. G. Cogan
Mathematics4 Nov 2026Peer-reviewedCC BY 4.0General mathematicsCrossref

Cite

BibTeX RIS

BibTeX

@misc{katoch2026accounting,
  title = {{Accounting for Bias Enables Sustainable LLM Evaluation}},
  author = {Harshita Katoch and David Antony Selby and Gerrit Großmann and Sebastian Vollmer},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.31184},
  eprint = {2609.31184},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.31184},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Accounting for Bias Enables Sustainable LLM Evaluation
AU  - Harshita Katoch
AU  - David Antony Selby
AU  - Gerrit Großmann
AU  - Sebastian Vollmer
PY  - 2026
DA  - 2026/09/25
PB  - arXiv
DO  - 10.48550/arxiv.2609.31184
UR  - https://arxiv.org/abs/2609.31184
AB  - LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.
ER  -

Abstract

Related

Cite