LLM Judge Validation Under Sparse Overlap: From Inference to Design
Abstract
Validating an LLM-as-a-judge requires estimating its agreement with humans, yet annotation budgets rarely allow every item to be multiply labeled. We prove that this overlap sparsity is the first-order determinant of wrong deployment decisions: at 5% pairwise overlap, wrong-decision rates reach 25% and the probability of selecting the wrong best judge among ten candidates is 65%. The two actionable levers are overlap quantity and allocation. For quantity, we derive a minimum-overlap formula showing ρ≥ 0.25 suffices for non-borderline judges while borderline cases remain fundamentally hard. For allocation, a zero-cost stratified scheme halves false-rejection rates relative to random sampling when strata are informative. We validate on 10 LLM judges across four evaluation matrices spanning visual assessment, causal reasoning, and summarization.
Related
- Scalable Sparse Autoregressive Variational Inference for Non-Gaussian Spatial Fields
- High-Dimensional Statistical Inference for Sparse Support Vector Machines
- Adaptive multi-resolution Gaussian processes: Scalable exact inference with naturally data-sparse covariance matrices
- Penalized Likelihood Inference in Highly Multivariate Sparse Spectral Graphical Gaussian Processes
- PSI-SINDy: Post-Selection Inference for Sparse Identification of Nonlinear Dynamics
BibTeX
@misc{li2026judge,
title = {{LLM Judge Validation Under Sparse Overlap: From Inference to Design}},
author = {Junxuan Li and Arko Mukherjee and Soumyabrata Pal},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.31857},
eprint = {2609.31857},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31857},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - LLM Judge Validation Under Sparse Overlap: From Inference to Design
AU - Junxuan Li
AU - Arko Mukherjee
AU - Soumyabrata Pal
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.31857
UR - https://arxiv.org/abs/2609.31857
AB - Validating an LLM-as-a-judge requires estimating its agreement with humans, yet annotation budgets rarely allow every item to be multiply labeled. We prove that this overlap sparsity is the first-order determinant of wrong deployment decisions: at 5% pairwise overlap, wrong-decision rates reach 25% and the probability of selecting the wrong best judge among ten candidates is 65%. The two actionable levers are overlap quantity and allocation. For quantity, we derive a minimum-overlap formula showing ρ≥ 0.25 suffices for non-borderline judges while borderline cases remain fundamentally hard. For allocation, a zero-cost stratified scheme halves false-rejection rates relative to random sampling when strata are informative. We validate on 10 LLM judges across four evaluation matrices spanning visual assessment, causal reasoning, and summarization.
ER -