Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
Abstract
Pairwise human-preference platforms such as Chatbot Arena have become central to large language model (LLM) evaluation, yet reliable task-specific ranking remains challenging. Global leaderboards mask task heterogeneity, while ranking each fine-grained task independently is unstable under sparse, imbalanced comparisons. We propose a low-rank framework for task-specific LLM ranking from sparse pairwise comparisons, modeling the task-by-model ability matrix Θ⋆ ∈ ℝdt × dm as low rank so that information is shared across related tasks while task-specific differences are preserved. We first develop a max-norm (ℓ∞) accurate estimator for the latent scores, combining a convex initializer with alternating-minimization refinement, and prove task-wise top-K recovery guarantees under sparse sampling. Our main contribution is an uncertainty quantification framework for task-specific ranking. We construct cross-fitted one-step debiased estimators for fixed score contrasts -- such as the task-specific ability gap between two models -- yielding asymptotically valid confidence intervals that attain the semiparametric efficiency bound. We then extend the inference to the high-dimensional ranking regime, where per-task ranks and top-K membership are determined by many dependent score-gap hypotheses. Using Gaussian and multiplier-bootstrap calibration, we obtain simultaneous confidence sets for per-task ranks and valid top-K membership tests across many tasks and models. Experiments on synthetic data and Chatbot Arena show that low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation and produces tighter, better-calibrated ranking certificates, with the largest gains in the sparse regime typical of real LLM benchmarks.
Related
- How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?
- Pairwise comparisons meet optimal transport theory
- From intersection names to street rankings in Kyoto: Comparing ranking methods for incomplete pairwise data
- Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons
- Entity Ranking with Covariates and Sparse Intrinsic Scores
BibTeX
@misc{li2026rank,
title = {{Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons}},
author = {Jiachun Li and David Simchi-Levi and Will Wei Sun},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.29395},
eprint = {2605.29395},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.29395},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
AU - Jiachun Li
AU - David Simchi-Levi
AU - Will Wei Sun
PY - 2026
DA - 2026/09/02
PB - arXiv
DO - 10.48550/arxiv.2605.29395
UR - https://arxiv.org/abs/2605.29395
AB - Pairwise human-preference platforms such as Chatbot Arena have become central to large language model (LLM) evaluation, yet reliable task-specific ranking remains challenging. Global leaderboards mask task heterogeneity, while ranking each fine-grained task independently is unstable under sparse, imbalanced comparisons. We propose a low-rank framework for task-specific LLM ranking from sparse pairwise comparisons, modeling the task-by-model ability matrix Θ^⋆ ∈ ℝ^d_t × d_m as low rank so that information is shared across related tasks while task-specific differences are preserved. We first develop a max-norm (ℓ_∞) accurate estimator for the latent scores, combining a convex initializer with alternating-minimization refinement, and prove task-wise top-K recovery guarantees under sparse sampling. Our main contribution is an uncertainty quantification framework for task-specific ranking. We construct cross-fitted one-step debiased estimators for fixed score contrasts -- such as the task-specific ability gap between two models -- yielding asymptotically valid confidence intervals that attain the semiparametric efficiency bound. We then extend the inference to the high-dimensional ranking regime, where per-task ranks and top-K membership are determined by many dependent score-gap hypotheses. Using Gaussian and multiplier-bootstrap calibration, we obtain simultaneous confidence sets for per-task ranks and valid top-K membership tests across many tasks and models. Experiments on synthetic data and Chatbot Arena show that low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation and produces tighter, better-calibrated ranking certificates, with the largest gains in the sparse regime typical of real LLM benchmarks.
ER -