Pair-Dependent Judge Errors Change the Minimax Rate of Semantic Entropy
Abstract
Semantic entropy, the entropy of the meanings among a language model's sampled answers, is estimated from the verdicts of a judge that declares pairs of answers equivalent or not, and automatic judges err in ways that depend on the answers they compare. The existing minimax theory covers only judges whose false-equivalence probability is the same for all pairs. We model a known judge that declares two answers with meanings a and b equivalent with probability q+ΔMab, for a symmetric matrix M and q bounded away from $0$ and $1$, and show that for three meanings the rate is set by tree statistics on four answers, while on an open set of channels the consistency threshold is set by trees on five. For every channel in an explicit open class, the minimax squared risk is of order N−1/3 with N=n4Δ6 on polynomial weak windows on which n5Δ8 exceeds a multiple of log n, and of order min{1,N−1/3} on every polynomial weak window when the tree functions on at most four vertices are injective. The exponent comes from a cubic degeneracy at the equitable point. On an open set of channels in the class, the four-vertex tree functions take equal values at two distributions with different entropies. For such a channel, on every polynomial weak window, the minimax risk is of the order of the sum of min{1,N−1/3} and a term that decays exponentially in n5Δ8, so no estimator is consistent while n5Δ8 stays bounded, even as N→∞. For Δ=n−β with β>1/2, channels of the class with injective four-vertex tree functions allow consistent estimation exactly for β<2/3, and the channels of this open set exactly for β<5/8, at which the risk exponent drops from 1/12 to $0$. Computer-assisted certificates show that both kinds of channels occur on open sets.
Related
- Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel
- Pollak's Minimax Quickest Change Detection: Non-Asymptotic Optimality
- Online Change-Point Detection in Linear Regression with Autoregressive Errors via Streaming Variational Bayes
- Coupled Usage-Sense Processes: Temporal and Attributable Lexical Semantic Change
- Minimax Additive Regression under Unknown Dependent Designs
BibTeX
@misc{liu2026pairdependent,
title = {{Pair-Dependent Judge Errors Change the Minimax Rate of Semantic Entropy}},
author = {Jiamiao Liu and Dewen Qiao and Yu Zhang and Xuetao Chen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39196},
eprint = {2610.39196},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39196},
}
RIS
TY - UNPB
TI - Pair-Dependent Judge Errors Change the Minimax Rate of Semantic Entropy
AU - Jiamiao Liu
AU - Dewen Qiao
AU - Yu Zhang
AU - Xuetao Chen
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39196
UR - https://arxiv.org/abs/2610.39196
AB - Semantic entropy, the entropy of the meanings among a language model's sampled answers, is estimated from the verdicts of a judge that declares pairs of answers equivalent or not, and automatic judges err in ways that depend on the answers they compare. The existing minimax theory covers only judges whose false-equivalence probability is the same for all pairs. We model a known judge that declares two answers with meanings a and b equivalent with probability q+ΔM_ab, for a symmetric matrix M and q bounded away from $0$ and $1$, and show that for three meanings the rate is set by tree statistics on four answers, while on an open set of channels the consistency threshold is set by trees on five. For every channel in an explicit open class, the minimax squared risk is of order N^−1/3 with N=n^4Δ^6 on polynomial weak windows on which n^5Δ^8 exceeds a multiple of log n, and of order min{1,N^−1/3} on every polynomial weak window when the tree functions on at most four vertices are injective. The exponent comes from a cubic degeneracy at the equitable point. On an open set of channels in the class, the four-vertex tree functions take equal values at two distributions with different entropies. For such a channel, on every polynomial weak window, the minimax risk is of the order of the sum of min{1,N^−1/3} and a term that decays exponentially in n^5Δ^8, so no estimator is consistent while n^5Δ^8 stays bounded, even as N→∞. For Δ=n^−β with β>1/2, channels of the class with injective four-vertex tree functions allow consistent estimation exactly for β<2/3, and the channels of this open set exactly for β<5/8, at which the risk exponent drops from 1/12 to $0$. Computer-assisted certificates show that both kinds of channels occur on open sets.
ER -