Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection
Abstract
Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after selection can omit selection cost. We study the conditional empirical Rademacher complexity of deep ReLU representations followed by bounded-norm predictors in additive or Lévy-Brownian RKHSs, termed Brownian heads. For a fixed representation, we derive an exact dual identity and sharp bounds in terms of activation mass, the average norm of the observed hidden vectors. Under same-sample selection, the representation supremum induces a quadratic Rademacher process. Brownian layer-cake and Gaussian-projection identities reduce it to coordinatewise or signed projected threshold traces, separating realized scale from selection complexity. For samples with pairwise-distinct inputs, explicit scalar ReLU families match the finite-trace and VC rates up to universal constants at the realized trace-and-envelope level. Induced-norm contraction also yields architecture-level bounds for rectangular, rank-deficient ReLU networks. Experiments verify the sharp bounds and rates, exhibit a selection gap at fixed activation mass, and assess the predictive feasibility of Brownian heads.
Related
- Sample genealogies within a Brownian excursion
- Exact finite-sample inference for multi-mixed fractional Brownian motion with drift
- When n is not enough: Assessing the interplay of spatial sampling error and sample selection in abundance estimation
- Collaborative representations for targeted causal inference under outcome selection
- A Composite Activation Function for Learning Stable Binary Representations
BibTeX
@misc{mohammadigohari2026brownian,
title = {{Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection}},
author = {Mahdi Mohammadigohari and Nicole Mücke},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.21422},
eprint = {2609.21422},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.21422},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection
AU - Mahdi Mohammadigohari
AU - Nicole Mücke
PY - 2026
DA - 2026/09/18
PB - arXiv
DO - 10.48550/arxiv.2609.21422
UR - https://arxiv.org/abs/2609.21422
AB - Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after selection can omit selection cost. We study the conditional empirical Rademacher complexity of deep ReLU representations followed by bounded-norm predictors in additive or Lévy-Brownian RKHSs, termed Brownian heads. For a fixed representation, we derive an exact dual identity and sharp bounds in terms of activation mass, the average norm of the observed hidden vectors. Under same-sample selection, the representation supremum induces a quadratic Rademacher process. Brownian layer-cake and Gaussian-projection identities reduce it to coordinatewise or signed projected threshold traces, separating realized scale from selection complexity. For samples with pairwise-distinct inputs, explicit scalar ReLU families match the finite-trace and VC rates up to universal constants at the realized trace-and-envelope level. Induced-norm contraction also yields architecture-level bounds for rectangular, rank-deficient ReLU networks. Experiments verify the sharp bounds and rates, exhibit a selection gap at fixed activation mass, and assess the predictive feasibility of Brownian heads.
ER -