Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects
Abstract
Large language models (LLMs) are increasingly used as judges for automated AI evaluation. A common practice is to randomize prompt sequences and average the resulting scores, but its statistical validity remains unclear. We show that LLM evaluation mechanisms can be approximated by a class of Markov generalized linear mixed models (GLMMs), supported by out-of-sample predictions across three major commercial LLMs. Using a first-order Markov GLMM, we study leaderboard ranking and group comparison. For leaderboard ranking, randomize-and-average selection is consistent under a mild separation condition, and a Williams square design can improve efficiency when item qualities are close. For group comparison, naive averaging can yield inconsistent conclusions about differences in group-level quality because of the response model's nonlinearity. Empirical results further support the validity of the proposed model-based inference beyond the first-order theory, including settings with higher-order sequence memory. We illustrate the approach in an application where AI judges compare two graphical model estimation methods.
Related
- Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt Instability
- BRACE: Blockwise Rank Aggregation for Correlation Estimation
- A reliability data aggregation method for fusion-oriented research facilities
- Sharp convergence rate of the estimation error of LASSO via the Feature Space Decomposition method
- Ground state energy estimation and preparation by the truncated power method
BibTeX
@misc{li2026trustworthy,
title = {{Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects}},
author = {Tianxi Li and Jie Ding},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07755},
eprint = {2610.07755},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07755},
}
RIS
TY - UNPB
TI - Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects
AU - Tianxi Li
AU - Jie Ding
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.07755
UR - https://arxiv.org/abs/2610.07755
AB - Large language models (LLMs) are increasingly used as judges for automated AI evaluation. A common practice is to randomize prompt sequences and average the resulting scores, but its statistical validity remains unclear. We show that LLM evaluation mechanisms can be approximated by a class of Markov generalized linear mixed models (GLMMs), supported by out-of-sample predictions across three major commercial LLMs. Using a first-order Markov GLMM, we study leaderboard ranking and group comparison. For leaderboard ranking, randomize-and-average selection is consistent under a mild separation condition, and a Williams square design can improve efficiency when item qualities are close. For group comparison, naive averaging can yield inconsistent conclusions about differences in group-level quality because of the response model's nonlinearity. Empirical results further support the validity of the proposed model-based inference beyond the first-order theory, including settings with higher-order sequence memory. We illustrate the approach in an application where AI judges compare two graphical model estimation methods.
ER -