What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control
Abstract
The information bottleneck (IB) trades compression of an input X against prediction of a target Y through a multiplier β=1/λ, yet each operating point is usually served by training, or fully conditioning, a new encoder. For jointly Gaussian data, we show that the IB solution factorizes as A(β)=D(β)W: the information-bearing directions W do not depend on β, and β acts only through scalar gains. We ask whether this separation of \emph{representation} from \emph{rate control} survives beyond Gaussian models, treating the budget as a given operating parameter rather than a quantity to be optimized. For a budget-free representation F(X), the \emph{separation gap} is controlled by the \emph{sufficiency deficit} εF=I(X;Y| F(X)): We show there is at least εF in the worst case for finite inputs, and it is bounded uniformly in λ by a function of εF for finite label alphabets. The same certificate extends to a new target distribution and to pooled multi-domain data, where a conditional-alignment term appears. On MNIST, SVHN, CIFAR-10 and STL-10 (10 seeds, eight prespecified budgets), a frozen CNN representation with one compressor--decoder pair per budget reduces estimated training multiply--accumulate operations (MACs) by 7.7× and wall-clock time by about 2× relative to end-to-end VIBs, and attains higher budget-averaged accuracy (65.6% versus 57.3% per domain; 67.1% versus 57.1% pooled), with the advantage concentrated at strong compression, where the VIBs collapse. At weak compression (λ≤0.036), VIB has higher cross-entropy and ECE; its surrogate Lagrangian is higher per domain, indistinguishable from the pooled VIB in the macro average, and lower on the pooled test set.
Related
- Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
- On the Information Bottleneck for Stable Variables
- Dynamics Aware Past-Future Information Bottleneck
- Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis
- On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck
BibTeX
@misc{tavakoli2026what,
title = {{What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control}},
author = {Hassan Tavakoli and Bella Bose},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36255},
eprint = {2610.36255},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36255},
}
RIS
TY - UNPB
TI - What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control
AU - Hassan Tavakoli
AU - Bella Bose
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.36255
UR - https://arxiv.org/abs/2610.36255
AB - The information bottleneck (IB) trades compression of an input X against prediction of a target Y through a multiplier β=1/λ, yet each operating point is usually served by training, or fully conditioning, a new encoder. For jointly Gaussian data, we show that the IB solution factorizes as A(β)=D(β)W: the information-bearing directions W do not depend on β, and β acts only through scalar gains. We ask whether this separation of \emph{representation} from \emph{rate control} survives beyond Gaussian models, treating the budget as a given operating parameter rather than a quantity to be optimized. For a budget-free representation F(X), the \emph{separation gap} is controlled by the \emph{sufficiency deficit} ε_F=I(X;Y| F(X)): We show there is at least ε_F in the worst case for finite inputs, and it is bounded uniformly in λ by a function of ε_F for finite label alphabets. The same certificate extends to a new target distribution and to pooled multi-domain data, where a conditional-alignment term appears. On MNIST, SVHN, CIFAR-10 and STL-10 (10 seeds, eight prespecified budgets), a frozen CNN representation with one compressor--decoder pair per budget reduces estimated training multiply--accumulate operations (MACs) by 7.7× and wall-clock time by about 2× relative to end-to-end VIBs, and attains higher budget-averaged accuracy (65.6% versus 57.3% per domain; 67.1% versus 57.1% pooled), with the advantage concentrated at strong compression, where the VIBs collapse. At weak compression (λ≤0.036), VIB has higher cross-entropy and ECE; its surrogate Lagrangian is higher per domain, indistinguishable from the pooled VIB in the macro average, and lower on the pooled test set.
ER -