Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
Abstract
LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.
Related
- Re-Ranking Methods for Instance-Level Image Retrieval: A Review
- Risk-Averse Decision Making with Multi-Level Reliability Guarantees
- A taxonomy of reliability indices for cluster-level constructs measured by multiple individual-level indicators.
- Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
- How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats
BibTeX
@misc{ba2026trust,
title = {{Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge}},
author = {Yang Ba and Michelle V. Mancenido and Rong Pan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.14029},
eprint = {2610.14029},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.14029},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
AU - Yang Ba
AU - Michelle V. Mancenido
AU - Rong Pan
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.14029
UR - https://arxiv.org/abs/2610.14029
AB - LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.
ER -