Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Abstract
An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the one-hidden-layer ReLU model of Awasthi et al. (COLT 2024). For width s, at most k active units per input, and effective weight and bias bounds W,B, every size-m sample in the class's fixed radius-R input domain satisfies ℛ(S)≤ CWRmin{k,√(sk/m)log3/2(2m)}+kB/√m. A support-preserving cover and a single normalized chaining argument remove the previous explicit dimension factor, up to logarithms. Lower bounds on appropriate i.i.d. marginals match up to those logarithms, showing how changing active units across inputs retains a width dependence. The input domain matters: zero-bias networks sparse on the entire ball have at most 2k nonzero units and complexity O(kWR/√m), whereas bias bounds comparable to WR restore the worst-case rate on that same domain in only logarithmic dimension. A spherical-cap construction proves the latter claim without assuming sparsity merely on the sampling support. For a specified normalized bounded loss and biases comparable to WR, we also obtain agnostic minimax excess-risk bounds of order min{1,√(s/(km))} up to logarithms.
Related
- Sharp p/log p Bounds for Metric Xp Inequalities and Arbitrary Rademacher Chaos
- Nearly Sharp Uniform Projection Bounds and Superlinear Vitushkin Inequalities for Deterministic Self-Similar Sets
- Prediction model for compressive strength of alkali-activated multi-source solid-waste grouting materials based on generative adversarial networks and machine learning
- Sparsely connected rank-inspired neural network
- The Rademacher quotient of L1 embeds into L1 and its weak Calkin algebra is uniformly incompressible
BibTeX
@misc{li2026nearly,
title = {{Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks}},
author = {Xiaoyu Li and Zhizhou Sha and Jiaojiao Jiang and Junbin Gao and Andi Han},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.09130},
eprint = {2609.09130},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09130},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
AU - Xiaoyu Li
AU - Zhizhou Sha
AU - Jiaojiao Jiang
AU - Junbin Gao
AU - Andi Han
PY - 2026
DA - 2026/09/08
PB - arXiv
DO - 10.48550/arxiv.2609.09130
UR - https://arxiv.org/abs/2609.09130
AB - An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the one-hidden-layer ReLU model of Awasthi et al. (COLT 2024). For width s, at most k active units per input, and effective weight and bias bounds W,B, every size-m sample in the class's fixed radius-R input domain satisfies ℛ(S)≤ CWRmin{k,√(sk/m)log^3/2(2m)}+kB/√m. A support-preserving cover and a single normalized chaining argument remove the previous explicit dimension factor, up to logarithms. Lower bounds on appropriate i.i.d. marginals match up to those logarithms, showing how changing active units across inputs retains a width dependence. The input domain matters: zero-bias networks sparse on the entire ball have at most 2k nonzero units and complexity O(kWR/√m), whereas bias bounds comparable to WR restore the worst-case rate on that same domain in only logarithmic dimension. A spherical-cap construction proves the latter claim without assuming sparsity merely on the sampling support. For a specified normalized bounded loss and biases comparable to WR, we also obtain agnostic minimax excess-risk bounds of order min{1,√(s/(km))} up to logarithms.
ER -