Human-Anchored Inference for Ranking New Models with Large Language Model Judges
Abstract
Human pairwise comparisons provide a reference for evaluating large language models (LLMs), but collecting sufficient judgments for each new release is costly and time-consuming. LLM judges offer a scalable alternative, although their comparisons may differ systematically from human preferences and across judges. We study the ranking of a new model that has received LLM-judge comparisons but no human comparisons. We propose ANCHOR (ANchored Comparisons for Human-reference inference with Orthogonal Riesz correction), which uses historical human and LLM comparisons to learn judge-specific sensitivities to human score differences and feature-dependent judge biases. These estimates are then used to infer the new model's human-reference score from its judge comparisons. The framework allows the feature distribution to change between historical and new-model comparisons. For inference, we construct a Neyman-orthogonal estimator through a joint Riesz correction that removes the first-order effects of estimating the historical human scores, judge sensitivities, and bias functions. We establish identification, convergence rates, and asymptotic normality with consistently estimable variance, and show that ANCHOR attains the semiparametric efficiency bound. Simulations demonstrate gains in score estimation and ranking accuracy. On Chatbot Arena, ANCHOR achieves the lowest score RMSE and insertion MAE among competing methods, with narrower score intervals on average.
Related
- How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats
- DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
- Towards Efficient Whole Page Ranking with One-Pass Inference
- SALTO: Solver-Anchored Lifted Transfer Operators for Nonlinear Event Inference
- SCOPE-Fact: Linguistic Scope-Guided Composition of Anchored Stochastic Operators for Predicting Event Factuality in Natural Language
BibTeX
@misc{zhou2026humananchored,
title = {{Human-Anchored Inference for Ranking New Models with Large Language Model Judges}},
author = {Xin Zhou and Sinian Zhang and Zhanyan Yang and Mingyuan Xu and Molei Liu and Doudou Zhou},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.19599},
eprint = {2609.19599},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19599},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Human-Anchored Inference for Ranking New Models with Large Language Model Judges
AU - Xin Zhou
AU - Sinian Zhang
AU - Zhanyan Yang
AU - Mingyuan Xu
AU - Molei Liu
AU - Doudou Zhou
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2609.19599
UR - https://arxiv.org/abs/2609.19599
AB - Human pairwise comparisons provide a reference for evaluating large language models (LLMs), but collecting sufficient judgments for each new release is costly and time-consuming. LLM judges offer a scalable alternative, although their comparisons may differ systematically from human preferences and across judges. We study the ranking of a new model that has received LLM-judge comparisons but no human comparisons. We propose ANCHOR (ANchored Comparisons for Human-reference inference with Orthogonal Riesz correction), which uses historical human and LLM comparisons to learn judge-specific sensitivities to human score differences and feature-dependent judge biases. These estimates are then used to infer the new model's human-reference score from its judge comparisons. The framework allows the feature distribution to change between historical and new-model comparisons. For inference, we construct a Neyman-orthogonal estimator through a joint Riesz correction that removes the first-order effects of estimating the historical human scores, judge sensitivities, and bias functions. We establish identification, convergence rates, and asymptotic normality with consistently estimable variance, and show that ANCHOR attains the semiparametric efficiency bound. Simulations demonstrate gains in score estimation and ranking accuracy. On Chatbot Arena, ANCHOR achieves the lowest score RMSE and insertion MAE among competing methods, with narrower score intervals on average.
ER -