Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt Instability

Abstract

LLM judges are often evaluated with a single prompt and only a few repeated calls. When their verdicts vary, it remains unclear whether the variation comes from sampling noise within a prompt or systematic differences across prompts. We formalize this distinction using a second-order response law: the distribution of prompt-conditioned verdict distributions induced by a declared prompt policy. For a quadratic measure of prompt instability, we show that the usual plug-in estimator is biased upward at finite repeat budgets because it confounds within-prompt noise with between-prompt variation. We derive unbiased estimators for both sampled prompts and declared fixed prompt censuses from the difference between within- and across-prompt agreement. Under a crossed prompt-by-answer-order design, the same framework separates prompt, order, interaction, and residual call variation, while retaining invalid completed outputs as outcomes. Known-law simulations and a byte-identical live null recover the predicted finite-R inflation. In a matched Qwen study, corrected low-repeat estimates are closer to an independently acquired R=16 reference than plug-in estimates, with the largest gains at small repeat budgets. A matched panel across four frozen judge configurations exhibits configuration-specific inflation magnitudes and component profiles. Prompt robustness can therefore be estimated separately from finite-call noise.

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
Zesheng Cai, Yingqi Fan, Sichang Chen +1
arXiv25 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
HEDE: Heritability estimation in high dimensions by ensembling debiased estimators in genome-wide association studies
Yanke Song, Xihong Lin, Pragya Sur
The Annals of Applied Statistics1 Sep 2026Peer-reviewedLicence not statedFull text12 citedStatisticsOpenAlex
Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects
Tianxi Li, Jie Ding
arXiv6 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Debiased inference for proximal dose-response function
Daeyoung Ham, Sihan Wu, Yifan Cui
arXiv7 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Second-order Wasserstein response for Lévy laws
Alexandre Autran
arXiv13 Oct 2026PreprintCC BY 4.0Full textOptimisation & control ProbabilityarXiv

Cite

BibTeX RIS

BibTeX

@misc{feng2026secondorder,
  title = {{Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt Instability}},
  author = {Pengbin Feng and Chunlei Meng and Daozheng Qu and Zhilin Zhang and Haoran Liu and Jiekai Wu},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2608.16253},
  eprint = {2608.16253},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.16253},
}

RIS

TY  - UNPB
TI  - Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt Instability
AU  - Pengbin Feng
AU  - Chunlei Meng
AU  - Daozheng Qu
AU  - Zhilin Zhang
AU  - Haoran Liu
AU  - Jiekai Wu
PY  - 2026
DA  - 2026/10/06
PB  - arXiv
DO  - 10.48550/arxiv.2608.16253
UR  - https://arxiv.org/abs/2608.16253
AB  - LLM judges are often evaluated with a single prompt and only a few repeated calls. When their verdicts vary, it remains unclear whether the variation comes from sampling noise within a prompt or systematic differences across prompts. We formalize this distinction using a second-order response law: the distribution of prompt-conditioned verdict distributions induced by a declared prompt policy. For a quadratic measure of prompt instability, we show that the usual plug-in estimator is biased upward at finite repeat budgets because it confounds within-prompt noise with between-prompt variation. We derive unbiased estimators for both sampled prompts and declared fixed prompt censuses from the difference between within- and across-prompt agreement. Under a crossed prompt-by-answer-order design, the same framework separates prompt, order, interaction, and residual call variation, while retaining invalid completed outputs as outcomes. Known-law simulations and a byte-identical live null recover the predicted finite-R inflation. In a matched Qwen study, corrected low-repeat estimates are closer to an independently acquired R=16 reference than plug-in estimates, with the largest gains at small repeat budgets. A matched panel across four frozen judge configurations exhibits configuration-specific inflation magnitudes and component profiles. Prompt robustness can therefore be estimated separately from finite-call noise.
ER  -

Abstract

Related

Cite