A Theory of Saddle Escape in Deep Nonlinear Networks

Abstract

In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of r layers initialized at the bottleneck scale ε, the escape time is Θ(Γ⁻¹) for r=1, Θ(Γ⁻¹log(1/ε)) for r=2, and Θ(Γ⁻¹ε^−(r−2)) for r≥3, where Γ=|𝔼[yx]|. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.

Wasserstein Saddle-Free Newton: Saddle Escape and Local Spectral Convergence
Razvan-Andrei Lascu, Taiji Suzuki
arXiv26 Sep 2026PreprintarXiv licenceFull textOptimisation & control StatisticsarXiv
Bayesian Quantile Deep Echo State Networks for Nonlinear Time Series
Antonio De Leon, Raquel Prado, Bruno Sansó
arXiv1 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv
Tikhonov-Regularized Dynamics and Inexact Algorithms for Nonlinear Saddle-Point Problems
Hong-lu Li, Yi-bin Xiao
arXiv29 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Linear Response Theory for Jump-Driven Stochastic Systems: Transient Statistics and Escape Dynamics
Qingyan Meng, Jinqiao Duan, Valerio Lucarini
arXiv14 Sep 2026PreprintarXiv licenceFull textDynamical systems ProbabilityarXiv
Convergence properties of nonlinear RLC electrical networks
Arjan van der Schaft
arXiv1 Nov 2026PreprintarXiv licenceFull textOptimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{rawal2026theory,
  title = {{A Theory of Saddle Escape in Deep Nonlinear Networks}},
  author = {Divit Rawal and Michael R. DeWeese},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2605.01288},
  eprint = {2605.01288},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.01288},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - A Theory of Saddle Escape in Deep Nonlinear Networks
AU  - Divit Rawal
AU  - Michael R. DeWeese
PY  - 2026
DA  - 2026/09/25
PB  - arXiv
DO  - 10.48550/arxiv.2605.01288
UR  - https://arxiv.org/abs/2605.01288
AB  - In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of r layers initialized at the bottleneck scale ε, the escape time is Θ(Γ^−1) for r=1, Θ(Γ^−1log(1/ε)) for r=2, and Θ(Γ^−1ε^−(r−2)) for r≥3, where Γ=|𝔼[yx]|. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.
ER  -

Abstract

Related

Cite