The Arbitrary-Placement Problem in Entropy-Minimizing Selection, and a Residual-Entropy Formulation
Abstract
Entropy-based selection objectives suffer from a fundamental degeneracy: minimizing Shannon entropy H(pA) rewards confident selection regardless of whether the selected candidate is informative. We address this limitation with the residual entropy D = H(pA) − H(pβ), where pβ is induced by candidate trust weights. We prove the exact identity D = −KL(pA‖ pβ) − Δ, where Δ measures whether the score-induced distribution and trust profile favor the same candidates. Boundary cases establish basic safety: under uniform trust, D≤0 automatically, so an equal-trust, non-starving state is never penalized, while at any one-hot limit, D→0 regardless of the selected candidate. For the intermediate regime where selection occurs, we prove that D≤0 when candidate ordering by trust agrees pairwise with ordering by informativeness, and derive a tighter certificate based on the leading candidate's margin over its competitors. These results are independent of the candidate-scoring function and apply to both stationary and dynamically changing information. Experiments with a gradient-based mixture-of-experts router confirm that the ordering conditions can hold during real optimization and show that correct ordering improves downstream performance when candidates are non-interchangeable and selections are used directly rather than averaged. Beyond routing, margin-based reweighting matches or outperforms fixed-strength baselines in a class-imbalance task, while informative selection in a production video-prediction system reduces MSE by approximately 20% and transfers to a related species. Residual entropy, therefore, provides a safety criterion for selection and a usable signal for deciding when that selection is informative.
Related
- A Purely Sequential Selection and Its Acceleration for the Best Treatment Under a New Formulation of the Indifference Zone: Asymptotic Theory and Simulations
- ARISE: An adaptive residual-informed stability ensemble for feature selection in small-sample biomedical omics
- On Simon's area-minimizing hypersurfaces with fractal singular sets
- Minimizing tangent maps into complex projective space
- Tetrahedral points and classification of Almgren-minimizing sets in ℝ3
BibTeX
@misc{shin2026arbitraryplacement,
title = {{The Arbitrary-Placement Problem in Entropy-Minimizing Selection, and a Residual-Entropy Formulation}},
author = {Alyssa H. Shin and Claire H. Shin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05925},
eprint = {2610.05925},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05925},
}
RIS
TY - UNPB
TI - The Arbitrary-Placement Problem in Entropy-Minimizing Selection, and a Residual-Entropy Formulation
AU - Alyssa H. Shin
AU - Claire H. Shin
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.05925
UR - https://arxiv.org/abs/2610.05925
AB - Entropy-based selection objectives suffer from a fundamental degeneracy: minimizing Shannon entropy H(p_A) rewards confident selection regardless of whether the selected candidate is informative. We address this limitation with the residual entropy D = H(p_A) − H(p_β), where p_β is induced by candidate trust weights. We prove the exact identity D = −KL(p_A‖ p_β) − Δ, where Δ measures whether the score-induced distribution and trust profile favor the same candidates. Boundary cases establish basic safety: under uniform trust, D≤0 automatically, so an equal-trust, non-starving state is never penalized, while at any one-hot limit, D→0 regardless of the selected candidate. For the intermediate regime where selection occurs, we prove that D≤0 when candidate ordering by trust agrees pairwise with ordering by informativeness, and derive a tighter certificate based on the leading candidate's margin over its competitors. These results are independent of the candidate-scoring function and apply to both stationary and dynamically changing information. Experiments with a gradient-based mixture-of-experts router confirm that the ordering conditions can hold during real optimization and show that correct ordering improves downstream performance when candidates are non-interchangeable and selections are used directly rather than averaged. Beyond routing, margin-based reweighting matches or outperforms fixed-strength baselines in a class-imbalance task, while informative selection in a production video-prediction system reduces MSE by approximately 20% and transfers to a related species. Residual entropy, therefore, provides a safety criterion for selection and a usable signal for deciding when that selection is informative.
ER -