Mutual Information Constrained Chernoff Bottleneck
Abstract
The classical information bottleneck (IB) measures the relevance of a representation U of X to a target Y by I(U;Y), which does not directly characterize the error of downstream decisions. For a binary hypothesis Y inferred from many separately encoded observations, the optimal error exponent is the Chernoff information between the two conditional distributions of U given Y. We study the mutual information constrained Chernoff bottleneck, which seeks an encoder that maximizes this Chernoff information subject to a rate constraint I(U;X) ≤ R. We show that its optimal value C(R) increases strictly up to R = H(V), where V merges the symbols of X with equal likelihood ratio, remains at the uncompressed exponent beyond, and, unlike the IB curve, need not be concave. We further show that k+1 outputs suffice to attain C(R), where k is the cardinality of V. We propose an alternating algorithm that updates the encoder via a generalized Blahut--Arimoto algorithm and the Chernoff parameter s via a nonlinear equation, and prove that its iterates remain feasible, with nondecreasing and convergent Chernoff information. Comparing with the classical IB for n samples, we show that, whenever the two objectives disagree, the IB encoder has the smaller equivocation H(Y|Un) at n = 1, whereas the Chernoff bottleneck has the smaller error probability and equivocation for all large n. Numerical experiments confirm the theory, and on real topic-detection data from the 20 Newsgroups corpus, compressing each word to only 17% of its entropy retains 90% of the error exponent and nearly the accuracy of the uncompressed classifier.
Related
- On the Information Bottleneck for Stable Variables
- Dynamics Aware Past-Future Information Bottleneck
- Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis
- On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck
- Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
BibTeX
@misc{tang2026mutual,
title = {{Mutual Information Constrained Chernoff Bottleneck}},
author = {Dier Tang and Guangyue Han},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.37994},
eprint = {2609.37994},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.37994},
}
RIS
TY - UNPB
TI - Mutual Information Constrained Chernoff Bottleneck
AU - Dier Tang
AU - Guangyue Han
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2609.37994
UR - https://arxiv.org/abs/2609.37994
AB - The classical information bottleneck (IB) measures the relevance of a representation U of X to a target Y by I(U;Y), which does not directly characterize the error of downstream decisions. For a binary hypothesis Y inferred from many separately encoded observations, the optimal error exponent is the Chernoff information between the two conditional distributions of U given Y. We study the mutual information constrained Chernoff bottleneck, which seeks an encoder that maximizes this Chernoff information subject to a rate constraint I(U;X) ≤ R. We show that its optimal value C(R) increases strictly up to R = H(V), where V merges the symbols of X with equal likelihood ratio, remains at the uncompressed exponent beyond, and, unlike the IB curve, need not be concave. We further show that k+1 outputs suffice to attain C(R), where k is the cardinality of V. We propose an alternating algorithm that updates the encoder via a generalized Blahut--Arimoto algorithm and the Chernoff parameter s via a nonlinear equation, and prove that its iterates remain feasible, with nondecreasing and convergent Chernoff information. Comparing with the classical IB for n samples, we show that, whenever the two objectives disagree, the IB encoder has the smaller equivocation H(Y|U^n) at n = 1, whereas the Chernoff bottleneck has the smaller error probability and equivocation for all large n. Numerical experiments confirm the theory, and on real topic-detection data from the 20 Newsgroups corpus, compressing each word to only 17% of its entropy retains 90% of the error exponent and nearly the accuracy of the uncompressed classifier.
ER -