Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
Abstract
Recent advances in multimodal foundation models have intensified the need to understand how different modalities share, preserve, and complement information. Mutual Information (MI), the Information Bottleneck (IB), and Partial Information Decomposition (PID) provide complementary perspectives, yet existing studies often treat them as isolated tools. This survey presents an information-theoretic perspective connecting these principles as progressively refined views of multimodal information processing: MI characterizes inter-modal dependency, IB explains task-oriented information preservation under compression, and PID decomposes preserved information into redundancy, uniqueness, and synergy. We review 170 recent studies (2018--2026) and 12 foundational works, organizing multimodal learning around four challenges: cross-modal alignment, information-efficient fusion, interaction-type characterization, and scaling to multimodal foundation models. Rather than using application domains as primary taxonomy axes, we interpret healthcare, robotics, recommendation systems, affective computing, and wireless communications as empirical validations of these information principles. Beyond taxonomy, we organize existing multimodal paradigms within a single information-theoretic coordinate system -- the Generalized Multimodal Information Lagrangian -- in which they occupy exact or approximate parameter corners, and whose unoccupied regions name candidate method families the literature has not yet built. We further discuss how emerging multimodal foundation models instantiate these principles at scale and identify open challenges including scalable information estimation in high-dimensional settings, standardized evaluation across information-theoretic methods, combinatorial complexity of multimodal PID, and the transition from post-hoc information analysis toward information-aware multimodal learning.
Related
- A Sample-Based Approach for Hierarchical Information-Theoretic Compression of Probabilistic Occupancy Grids
- When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting
- Learning Rules, Memorizing Facts: An Information-Theoretic View of Hallucination
- Efficient Robust Learning at the Information-Theoretic Limit
- Fundamental Limits of Information-Theoretic Secure Aggregation with User Dropouts in Federated Learning
BibTeX
@misc{wen2026dependency,
title = {{Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning}},
author = {Liangjian Wen and Linjie Li and Jiang Duan and Yong Dai and Jianzhuang Liu and Zhao Kang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.14421},
eprint = {2609.14421},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.14421},
note = {CC0},
}
RIS
TY - UNPB
TI - Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
AU - Liangjian Wen
AU - Linjie Li
AU - Jiang Duan
AU - Yong Dai
AU - Jianzhuang Liu
AU - Zhao Kang
PY - 2026
DA - 2026/09/13
PB - arXiv
DO - 10.48550/arxiv.2609.14421
UR - https://arxiv.org/abs/2609.14421
AB - Recent advances in multimodal foundation models have intensified the need to understand how different modalities share, preserve, and complement information. Mutual Information (MI), the Information Bottleneck (IB), and Partial Information Decomposition (PID) provide complementary perspectives, yet existing studies often treat them as isolated tools. This survey presents an information-theoretic perspective connecting these principles as progressively refined views of multimodal information processing: MI characterizes inter-modal dependency, IB explains task-oriented information preservation under compression, and PID decomposes preserved information into redundancy, uniqueness, and synergy. We review 170 recent studies (2018--2026) and 12 foundational works, organizing multimodal learning around four challenges: cross-modal alignment, information-efficient fusion, interaction-type characterization, and scaling to multimodal foundation models. Rather than using application domains as primary taxonomy axes, we interpret healthcare, robotics, recommendation systems, affective computing, and wireless communications as empirical validations of these information principles. Beyond taxonomy, we organize existing multimodal paradigms within a single information-theoretic coordinate system -- the Generalized Multimodal Information Lagrangian -- in which they occupy exact or approximate parameter corners, and whose unoccupied regions name candidate method families the literature has not yet built. We further discuss how emerging multimodal foundation models instantiate these principles at scale and identify open challenges including scalable information estimation in high-dimensional settings, standardized evaluation across information-theoretic methods, combinatorial complexity of multimodal PID, and the transition from post-hoc information analysis toward information-aware multimodal learning.
ER -