PreprintarXiv licencearXiv
Para-Gaussian Approximations of Random Deep Neural Networks
Abstract
It is known that randomly initialized neural networks have Gaussian behaviour in the n→ ∞ limit. Unfortunately, this approximation can be poor for finite n and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is \emph{slightly} non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).
Related
- Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks
- Depth Separation for Neural Networks under Gaussian Measure
- Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence
- A Smoothed Discrepancy Principle for Random Feature Methods and Neural Networks
- Constructive Universal Approximation by Infinite-Dimensional Neural Networks via Gaussian-Weighted Ridgelet Transform
BibTeX
@misc{nica2026paragaussian,
title = {{Para-Gaussian Approximations of Random Deep Neural Networks}},
author = {Mihai Nica and Janosch Ortmann},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.37388},
eprint = {2610.37388},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.37388},
}
RIS
TY - UNPB
TI - Para-Gaussian Approximations of Random Deep Neural Networks
AU - Mihai Nica
AU - Janosch Ortmann
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.37388
UR - https://arxiv.org/abs/2610.37388
AB - It is known that randomly initialized neural networks have Gaussian behaviour in the n→ ∞ limit. Unfortunately, this approximation can be poor for finite n and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is \emph{slightly} non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).
ER -