Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability
Abstract
Modern deep learning has been shown to operate using learning rates far larger than those justified by classical optimization theory. Most prior analyses of this phenomenon focus on deterministic gradient descent, leaving the stochastic setting largely unexplored. In this work, we provide sharp convergence guarantees for Stochastic Gradient Descent (SGD) applied to the multiclass cross-entropy loss, for both linear classifiers and two-layer neural networks. We show that the stochasticity of SGD may cause the dynamics to alternate between an edge-of-stability regime that is dominated by curvature-driven oscillations, and a stable regime in which the expected loss decreases at a controlled rate. Despite that, we prove that SGD stabilizes the dynamics, ensuring that the iterates return to stability in a fixed number of iterations and allowing convergence in the best-iterate sense even with large learning rates. Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime.
Related
- Renyi's Relative Entropy Lyapunov Functional for Convergence to Invariant Measure in Dissipative Stochastic Hamiltonian Systems
- A Shadow Sequence Technique for Last-Iterate Convergence Analysis of the Random Reshuffling Method: Relaxed Stepsize Conditions and Beyond
- Lyapunov Functions and R-Linear Convergence for Quadratic Barzilai-Borwein Dynamics
- Local Lyapunov Exponent-Guided Stochastic Midcourse Correction Design of Low-Energy Earth-Moon Transfers
- Lyapunov stability of polynomial vector fields is undecidable
BibTeX
@misc{emmanouilidis2026convergence,
title = {{Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability}},
author = {Konstantinos Emmanouilidis and Lachlan MacDonald and Salma Tarmoun and Rene Vidal},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.30930},
eprint = {2606.30930},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.30930},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability
AU - Konstantinos Emmanouilidis
AU - Lachlan MacDonald
AU - Salma Tarmoun
AU - Rene Vidal
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2606.30930
UR - https://arxiv.org/abs/2606.30930
AB - Modern deep learning has been shown to operate using learning rates far larger than those justified by classical optimization theory. Most prior analyses of this phenomenon focus on deterministic gradient descent, leaving the stochastic setting largely unexplored. In this work, we provide sharp convergence guarantees for Stochastic Gradient Descent (SGD) applied to the multiclass cross-entropy loss, for both linear classifiers and two-layer neural networks. We show that the stochasticity of SGD may cause the dynamics to alternate between an edge-of-stability regime that is dominated by curvature-driven oscillations, and a stable regime in which the expected loss decreases at a controlled rate. Despite that, we prove that SGD stabilizes the dynamics, ensuring that the iterates return to stability in a fixed number of iterations and allowing convergence in the best-iterate sense even with large learning rates. Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime.
ER -