Critical initialization destabilizes higher input derivatives in wide scalar-input networks

Abstract

The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.

Optimal Initialization Scale for Neural Networks with Locally Quadratic Loss Landscapes: An SGD Dynamics Perspective
Hiroshi Horii, Sothea Has
IEEE Transactions on Neural Networks and Learning Systems17 Aug 2026Peer-reviewedPublisher termsFull textStatisticsHAL
Activation-Matched Gibbs Initialization for Neural Networks
Diego Pinto-Suárez, Jordan Patracone, Marc Sebban
HAL19 Oct 2026PreprintCC BY 4.0Full textStatisticsHAL
Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
Alexandre Declèves, Etienne Boursier, Nicolas Flammarion
arXiv28 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
Correlated initialization of deep residual networks
Felix Benning, Ivan Nourdin, Giovanni Peccati
arXiv3 Sep 2026PreprintCC BY-SA 4.0Full textProbability StatisticsarXiv
Higher derivatives of the Gram function
Marius Dundulis, Ramūnas Garunkštis, Antanas Laurinčikas +1
arXiv12 Oct 2026PreprintCC BY 4.0Full textAlgebra & number theoryarXiv

Cite

BibTeX RIS

BibTeX

@misc{singh2026critical,
  title = {{Critical initialization destabilizes higher input derivatives in wide scalar-input networks}},
  author = {Prashant Singh and Pranav Singh},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.09244},
  eprint = {2609.09244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.09244},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Critical initialization destabilizes higher input derivatives in wide scalar-input networks
AU  - Prashant Singh
AU  - Pranav Singh
PY  - 2026
DA  - 2026/09/08
PB  - arXiv
DO  - 10.48550/arxiv.2609.09244
UR  - https://arxiv.org/abs/2609.09244
AB  - The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.
ER  -

Abstract

Related

Cite