The injectivity threshold of a Gaussian ReLU layer
Abstract
We determine how wide a randomly initialized ReLU layer must be to distinguish every pair of inputs. For independent Gaussian weights and zero bias, as the dimensions grow at a fixed output-to-input ratio, the probability of global injectivity tends to zero below a sharp threshold and to one above it. We characterize this threshold exactly through a variational formula for the limiting minimum number of active neurons per input coordinate. The formula accounts for correlations among input directions at arbitrarily many levels. We prove this formula rigorously, establishing a prediction from the statistical physics of the spherical perceptron. The proof compares systems of nearby dimensions and controls a Gaussian interpolation by calibrating responses to auxiliary fields. A uniform volume estimate connects the soft minimum to the worst input direction. An exact-arithmetic calculation places the threshold below seven output neurons per input coordinate. Numerical evaluations suggest a value near 6.698.
Related
- Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
- Two-Layer Random ReLU Networks Can Separate Disjoint Classes
- Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas
- Boundary-layer asymptotics for Gaussian-smoothed singular measures
- Gaussian Critical-Threshold Instability in Real Phase Retrieval
BibTeX
@misc{xie2026injectivity,
title = {{The injectivity threshold of a Gaussian ReLU layer}},
author = {Xiaohui Xie},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07492},
eprint = {2610.07492},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07492},
}
RIS
TY - UNPB
TI - The injectivity threshold of a Gaussian ReLU layer
AU - Xiaohui Xie
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.07492
UR - https://arxiv.org/abs/2610.07492
AB - We determine how wide a randomly initialized ReLU layer must be to distinguish every pair of inputs. For independent Gaussian weights and zero bias, as the dimensions grow at a fixed output-to-input ratio, the probability of global injectivity tends to zero below a sharp threshold and to one above it. We characterize this threshold exactly through a variational formula for the limiting minimum number of active neurons per input coordinate. The formula accounts for correlations among input directions at arbitrarily many levels. We prove this formula rigorously, establishing a prediction from the statistical physics of the spherical perceptron. The proof compares systems of nearby dimensions and controls a Gaussian interpolation by calibrating responses to auxiliary fields. A uniform volume estimate connects the soft minimum to the worst input direction. An exact-arithmetic calculation places the threshold below seven output neurons per input coordinate. Numerical evaluations suggest a value near 6.698.
ER -