Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis

Abstract

Which pretraining objective yields the best representation for downstream tasks under an information budget? We formalize this problem as a multitask information bottleneck, using a mutual information constraint to model incomplete information retention. For Gaussian inputs and tasks, we characterize optimal representations and show that a weakly task-aligned specialist can outperform a reconstruction generalist at low rates but lose at high rates. We also construct task families for which every fixed Gaussian prediction objective incurs arbitrarily large relative regret, driven by reversals in optimal allocation priorities. Conversely, the average task information matrix defines an objective with a uniform relative-regret bound controlled by maximum task SNR. CelebA and dSprites experiments illustrate that compression can change which pretraining objective performs best, and even task-aware training can reverse which downstream task is better preserved as rate increases.

On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck
Dier Tang, Jun Chen
arXiv5 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
On the Information Bottleneck for Stable Variables
Dongmin Park, Jihad Fahs, Mohamad Assaad
arXiv18 Oct 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv
Dynamics Aware Past-Future Information Bottleneck
Wooyoung Chung, Stas Tiomkin
arXiv5 Nov 2026PreprintarXiv licenceFull textGeneral mathematicsarXiv
Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
Meng'en Qin, Yinchen Liu, Mingxuan Cui +1
arXiv27 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv
On the Information Bottleneck for Ergodic Finite-State Markov Chains: Stationary Approximation and Asymptotic Optimality
Wael Tabbara, Jihad Fahs, Mohamad Assaad
arXiv19 Oct 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv

Cite

BibTeX RIS

BibTeX

@misc{stiles2026optimal,
  title = {{Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis}},
  author = {Nicholas Stiles and Gregory W. Wornell},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.30493},
  eprint = {2610.30493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30493},
}

RIS

TY  - UNPB
TI  - Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis
AU  - Nicholas Stiles
AU  - Gregory W. Wornell
PY  - 2026
DA  - 2026/10/21
PB  - arXiv
DO  - 10.48550/arxiv.2610.30493
UR  - https://arxiv.org/abs/2610.30493
AB  - Which pretraining objective yields the best representation for downstream tasks under an information budget? We formalize this problem as a multitask information bottleneck, using a mutual information constraint to model incomplete information retention. For Gaussian inputs and tasks, we characterize optimal representations and show that a weakly task-aligned specialist can outperform a reconstruction generalist at low rates but lose at high rates. We also construct task families for which every fixed Gaussian prediction objective incurs arbitrarily large relative regret, driven by reversals in optimal allocation priorities. Conversely, the average task information matrix defines an objective with a uniform relative-regret bound controlled by maximum task SNR. CelebA and dSprites experiments illustrate that compression can change which pretraining objective performs best, and even task-aware training can reverse which downstream task is better preserved as rate increases.
ER  -

Abstract

Related

Cite