Convergence of stochastic gradient descent for learning linear neural networks
Abstract
Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.
Related
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
- Convergence rate of stochastic gradient descent under Łojasiewicz type condition and polynomially controlled noise
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
- On the convergence of doubly stochastic block gradient descent method for linear inverse problems
BibTeX
@misc{nguegnang2026convergence,
title = {{Convergence of stochastic gradient descent for learning linear neural networks}},
author = {Gabin Maxime Nguegnang and Bubacarr Bah and Holger Rauhut and Ulrich Terstiege},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28561},
eprint = {2610.28561},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28561},
}
RIS
TY - UNPB
TI - Convergence of stochastic gradient descent for learning linear neural networks
AU - Gabin Maxime Nguegnang
AU - Bubacarr Bah
AU - Holger Rauhut
AU - Ulrich Terstiege
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28561
UR - https://arxiv.org/abs/2610.28561
AB - Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.
ER -