The Dynamics of RMSProp Beyond the Stability Threshold
Abstract
We study what happens to RMSProp when the learning rate is too large for convergence to the minimizer. For positive-definite quadratic loss functions, we show that the minimizer is Lyapunov un- stable above an explicit critical learning rate. Nevertheless, every trajectory eventually enters a compact trapping region. We obtain explicit bounds proportional to h in the spatial variables and to h2 in the memory variables, where h is the learning rate. We also prove that nontrivial period-two orbits exist precisely above the critical learning rate, and under additional spectral assumptions, an attracting branch of these orbits emerges from the minimizer. For sufficiently large learning rates, we construct branches near admissible corners of a cube. We then explore the dynamics inside the trapping region using the parameterization method and numerical continuation, exploring invariant circles and tori, coexistence of attracting sets, circle-doubling sequences, and chaotic motions. We examine how the dynamics depend on the loss Hessian, memory decay, and dimension. Finally, we test a learning-rate reduction schedule and continue selected invariant circles into an autonomous variant of the Adam optimizer.
Related
- On the Sobolev stability threshold for 3D Navier-Stokes equations with rotation near the Couette flow
- Lie bracket approximations of RMSprop for Extremum Seeking Control
- Probability threshold for the stability of $α(G(n,r,s))$
- Enhanced dissipation and stability threshold for the Navier--Stokes equations near Poiseuille flow
- Stability threshold of the Poiseuille flow in 2D Navier-Stokes Boussinesq equations
BibTeX
@misc{loh2026dynamics,
title = {{The Dynamics of RMSProp Beyond the Stability Threshold}},
author = {Kyle Loh and J. D. Mireles James and Zhenghao Xu and Molei Tao},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30078},
eprint = {2610.30078},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30078},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Dynamics of RMSProp Beyond the Stability Threshold
AU - Kyle Loh
AU - J. D. Mireles James
AU - Zhenghao Xu
AU - Molei Tao
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.30078
UR - https://arxiv.org/abs/2610.30078
AB - We study what happens to RMSProp when the learning rate is too large for convergence to the minimizer. For positive-definite quadratic loss functions, we show that the minimizer is Lyapunov un- stable above an explicit critical learning rate. Nevertheless, every trajectory eventually enters a compact trapping region. We obtain explicit bounds proportional to h in the spatial variables and to h2 in the memory variables, where h is the learning rate. We also prove that nontrivial period-two orbits exist precisely above the critical learning rate, and under additional spectral assumptions, an attracting branch of these orbits emerges from the minimizer. For sufficiently large learning rates, we construct branches near admissible corners of a cube. We then explore the dynamics inside the trapping region using the parameterization method and numerical continuation, exploring invariant circles and tori, coexistence of attracting sets, circle-doubling sequences, and chaotic motions. We examine how the dynamics depend on the loss Hessian, memory decay, and dimension. Finally, we test a learning-rate reduction schedule and continue selected invariant circles into an autonomous variant of the Adam optimizer.
ER -