Intrinsic Information-Theoretic Analysis of ReLU Nets
Abstract
Mutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters whose bias often dominates the signal. We show that continuous piecewise-linear (CPWL) networks, including ReLU networks, admit an intrinsic information-theoretic analysis that avoids this pitfall: their inherent partition of the input space into convex linear regions assigns to every input a discrete region label Π alongside the continuous representation T and label Y. We introduce a decomposition of the mutual information I(Y;T) into interpretable geometric terms. We show that the dominant term, the *routing information* $I(Y; Π)$, admits an estimable plug-in estimator, that in the well-sampled regime, approximates a population lower bound on the joint information $I(Y; Π, T)$. When the partition saturates, a functional-equivalence quotient collapses regions implementing the same linear operator and restores informativeness; this extension has a single bounded, dimensionless parameter, which is reported through the quotient's compression ratio. The routing estimator agrees strongly with three established MI baselines applied to the same networks and data. The functional quotient recovers informative estimates in the regime bordering the well-sampled and under-sampled regime. We provide a framework for analysing the internal information geometry of CPWL networks, grounded in their inherent piecewise-linear structure rather than in external discretisation.
Related
- Measuring Learned Equivariance: An Information Theoretic Approach
- A Sample-Based Approach for Hierarchical Information-Theoretic Compression of Probabilistic Occupancy Grids
- Conditional Fisher-Information Central Limit Theorems under Log-Concavity with Information-Theoretic Consequences
- Learning Rules, Memorizing Facts: An Information-Theoretic View of Hallucination
- Efficient Robust Learning at the Information-Theoretic Limit
BibTeX
@misc{myliuskroken2026intrinsic,
title = {{Intrinsic Information-Theoretic Analysis of ReLU Nets}},
author = {Johan Mylius-Kroken and Elisabeth Wetzer and Ali Ramezani-Kebrya and Robert Jenssen and Kristoffer Knutsen Wickstrøm},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36864},
eprint = {2610.36864},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36864},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Intrinsic Information-Theoretic Analysis of ReLU Nets
AU - Johan Mylius-Kroken
AU - Elisabeth Wetzer
AU - Ali Ramezani-Kebrya
AU - Robert Jenssen
AU - Kristoffer Knutsen Wickstrøm
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.36864
UR - https://arxiv.org/abs/2610.36864
AB - Mutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters whose bias often dominates the signal. We show that continuous piecewise-linear (CPWL) networks, including ReLU networks, admit an intrinsic information-theoretic analysis that avoids this pitfall: their inherent partition of the input space into convex linear regions assigns to every input a discrete region label Π alongside the continuous representation T and label Y. We introduce a decomposition of the mutual information I(Y;T) into interpretable geometric terms. We show that the dominant term, the *routing information* $I(Y; Π)$, admits an estimable plug-in estimator, that in the well-sampled regime, approximates a population lower bound on the joint information $I(Y; Π, T)$. When the partition saturates, a functional-equivalence quotient collapses regions implementing the same linear operator and restores informativeness; this extension has a single bounded, dimensionless parameter, which is reported through the quotient's compression ratio. The routing estimator agrees strongly with three established MI baselines applied to the same networks and data. The functional quotient recovers informative estimates in the regime bordering the well-sampled and under-sampled regime. We provide a framework for analysing the internal information geometry of CPWL networks, grounded in their inherent piecewise-linear structure rather than in external discretisation.
ER -