Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel
Abstract
The semantic entropy of a language model is the entropy of the meanings of its sampled answers. The meanings are never observed, and the entropy is estimated from the verdicts of a pairwise equivalence judge. Given the meanings of n answers, a known channel declares each pair equivalent independently, with probability q+Δ within a meaning and q across meanings. At most K meanings occur, and q is bounded away from $0$ and $1$. For a weak judge, with nΔ2→0, the global minimax rate is set by distributions near a vertex of the simplex and overstates the difficulty elsewhere. We show that the difficulty is set by how many meanings can be rare. Over distributions with at most s meanings below a fixed probability τ<1/(K−s), the minimax squared risk is of order N−1/s for s≥2, N−1log2(e+N) for s=1 and N−1 for s=0, capped at one, where N=nKΔ2K−2. The matching lower bounds hold on every fixed polynomial window nΔ2≤ n−κ. One estimator attains these rates without knowing s or τ: it averages products of centered verdicts along all spanning trees of every k-set of answers, projects the resulting power sums onto the valid moment set and inverts them. The case s=K−1 gives the global rate N−1/(K−1) and the consistency threshold: with Δ=n−γ the entropy is uniformly consistently estimable exactly when γ<K/(2K−2). On polynomial windows, the same estimator attains the sharp local minimax constant on shrinking neighborhoods of interior points with distinct probabilities. Restricting the average to stars multiplies the constant by KK−3.
Related
- Pair-Dependent Judge Errors Change the Minimax Rate of Semantic Entropy
- Adaptive pairwise preference elicitation for decision support: identifying the best alternative with fewer judgments
- A Single Physical Principle Determines Quantum Relative Entropy
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments
- Analyzing Binary Judgments: An Empirical and Simulation-Based Comparison of Aggregated and Trial-Level Approaches for Hypothesis Testing
BibTeX
@misc{yu2026rarity,
title = {{Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel}},
author = {Xiaoli Yu and Jiamiao Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17971},
eprint = {2610.17971},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17971},
}
RIS
TY - UNPB
TI - Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel
AU - Xiaoli Yu
AU - Jiamiao Liu
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.17971
UR - https://arxiv.org/abs/2610.17971
AB - The semantic entropy of a language model is the entropy of the meanings of its sampled answers. The meanings are never observed, and the entropy is estimated from the verdicts of a pairwise equivalence judge. Given the meanings of n answers, a known channel declares each pair equivalent independently, with probability q+Δ within a meaning and q across meanings. At most K meanings occur, and q is bounded away from $0$ and $1$. For a weak judge, with nΔ^2→0, the global minimax rate is set by distributions near a vertex of the simplex and overstates the difficulty elsewhere. We show that the difficulty is set by how many meanings can be rare. Over distributions with at most s meanings below a fixed probability τ<1/(K−s), the minimax squared risk is of order N^−1/s for s≥2, N^−1log^2(e+N) for s=1 and N^−1 for s=0, capped at one, where N=n^KΔ^2K−2. The matching lower bounds hold on every fixed polynomial window nΔ^2≤ n^−κ. One estimator attains these rates without knowing s or τ: it averages products of centered verdicts along all spanning trees of every k-set of answers, projects the resulting power sums onto the valid moment set and inverts them. The case s=K−1 gives the global rate N^−1/(K−1) and the consistency threshold: with Δ=n^−γ the entropy is uniformly consistently estimable exactly when γ<K/(2K−2). On polynomial windows, the same estimator attains the sharp local minimax constant on shrinking neighborhoods of interior points with distinct probabilities. Restricting the average to stars multiplies the constant by K^K−3.
ER -