OSCAR: Order-aware Sensitivity Correction for AI Rankings
Abstract
LLM judges evaluate candidate language models through pairwise comparisons of their responses. Judge-aware Bradley--Terry models aggregate pairwise comparisons by jointly estimating candidate scores and judge-specific sensitivities (how strongly each judge responds to quality differences), often interpreted as reliability. We show that this reading fails when judges prefer a display position. Even under balanced display orders or swap-and-average, omitting position attenuates fitted judge sensitivity by approximately the factor 1−D2 near equal quality, where D is the judge's net first-slot preference between equally good responses. We propose OSCAR, which jointly estimates candidate scores, judge sensitivities and judge-specific position, length and family effects, and provides confidence intervals for pairwise win probabilities that account for dependence within prompts. On held-out data, observed attenuation closely matches the prediction 1−D2 (correlation of log ratios 0.99). Across four datasets, position adjustment lowers held-out log loss by 14-23% and changes which judges would be dropped for low sensitivity on three datasets. On MT-Bench, it ranks GPT-4 first instead of claude-v1, matching human votes. Simulations with known parameters show improved recovery of judge sensitivities and candidate rankings, with 94.4-95.2% confidence interval coverage. Low estimated sensitivity alone is therefore not evidence of an unreliable judge.
Related
- Edge-Defect Spectral Methods for Higher-Order Rankings of Spanning Tree Counts
- A Geometry-Aware Residual Correction of Hagan's SABR Implied Volatility Formula
- Third-order Halley-type iterative method with positive and bounded correction function
- Adaptive Reliability-Aware Nonlinear State Estimation Framework for GNSS/INS Integrated Navigation with Bounded Position Correction
- High-order positivity-preserving well-balanced schemes: combining flux globalization with modified Patankar deferred correction methods
BibTeX
@misc{liu2026oscar,
title = {{OSCAR: Order-aware Sensitivity Correction for AI Rankings}},
author = {You Liu and Yue Liu and Quanchao Lu and Ergan Shang and Nick Shipilov},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.24128},
eprint = {2609.24128},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24128},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - OSCAR: Order-aware Sensitivity Correction for AI Rankings
AU - You Liu
AU - Yue Liu
AU - Quanchao Lu
AU - Ergan Shang
AU - Nick Shipilov
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2609.24128
UR - https://arxiv.org/abs/2609.24128
AB - LLM judges evaluate candidate language models through pairwise comparisons of their responses. Judge-aware Bradley--Terry models aggregate pairwise comparisons by jointly estimating candidate scores and judge-specific sensitivities (how strongly each judge responds to quality differences), often interpreted as reliability. We show that this reading fails when judges prefer a display position. Even under balanced display orders or swap-and-average, omitting position attenuates fitted judge sensitivity by approximately the factor 1−D^2 near equal quality, where D is the judge's net first-slot preference between equally good responses. We propose OSCAR, which jointly estimates candidate scores, judge sensitivities and judge-specific position, length and family effects, and provides confidence intervals for pairwise win probabilities that account for dependence within prompts. On held-out data, observed attenuation closely matches the prediction 1−D^2 (correlation of log ratios 0.99). Across four datasets, position adjustment lowers held-out log loss by 14-23% and changes which judges would be dropped for low sensitivity on three datasets. On MT-Bench, it ranks GPT-4 first instead of claude-v1, matching human votes. Simulations with known parameters show improved recovery of judge sensitivities and candidate rankings, with 94.4-95.2% confidence interval coverage. Low estimated sensitivity alone is therefore not evidence of an unreliable judge.
ER -