LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability
Abstract
Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.
Related
- The impact of spin priors on parameterized tests of general relativity
- Super-Resolution of Solar Magnetograms via Adaptive Stratified Ensemble Learning with Uncertainty Estimation
- On the Interpretation of Tests of the Horizon Area Law
- Jeans criterion in hydrostatic stratified media
- Local tests of Schwarzschild-Bertotti-Robinson spacetime
BibTeX
@misc{usami2026judges,
title = {{LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability}},
author = {Hiroyasu Usami and Keisuke Hara and Ayato Tsuboi and Naohiko Matsuda},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.15610},
eprint = {2606.15610},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.15610},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability
AU - Hiroyasu Usami
AU - Keisuke Hara
AU - Ayato Tsuboi
AU - Naohiko Matsuda
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2606.15610
UR - https://arxiv.org/abs/2606.15610
AB - Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.
ER -