DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
Abstract
Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.
Related
- Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt Instability
- Adaptive Preference Elicitation in Preference Robust CPT-Based Shortfall
- Adaptive pairwise preference elicitation for decision support: identifying the best alternative with fewer judgments
- Adaptive Reliability-Aware Nonlinear State Estimation Framework for GNSS/INS Integrated Navigation with Bounded Position Correction
- A Path-Dependent Agent-Based Microsimulation of Crime and Violence Reduction Policy Portfolios in Bolivia: Survey Calibration, Adaptive Emulator Ensembles, Global Sensitivity, Tempered MCMC, and Multi-Objective Policy Analysis
BibTeX
@misc{cai2026dial,
title = {{DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration}},
author = {Zesheng Cai and Yingqi Fan and Sichang Chen and Jin-Hong Du},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.31215},
eprint = {2609.31215},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31215},
}
RIS
TY - UNPB
TI - DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
AU - Zesheng Cai
AU - Yingqi Fan
AU - Sichang Chen
AU - Jin-Hong Du
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.31215
UR - https://arxiv.org/abs/2609.31215
AB - Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.
ER -