Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions
Abstract
Existing theory of momentum assumes that gradients arrive at every parameter at a roughly constant rate, an assumption violated in practice by heavy-tailed data distributions and modern architectures. We theoretically analyze the dynamics of two tractable models of momentum under sparse updates: a least squares model with sparse inputs and a logistic regression model with a rare class. Both admit exact closed-form second-moment dynamics whose high-dimensional limits we characterize across three scaling exponents for sparsity, batch size, and momentum decay. The phase structure on both problems is governed by the ratio of two intrinsic timescales: a momentum retention timescale (how many active updates the buffer survives) and a learning timescale (how many active updates it takes to reduce the squared error). When learning is much slower than retention, the limit matches SGD; when learning is faster, the system is unstable; where the timescales coincide, we recover classical heavy-ball dynamics. The oscillatory dynamics occur at different momentum values for different token sparsity, creating a spectral conflict for global momentum across token frequencies.
Related
- Accelerated Push-Pull with Local Updates for Decentralized Stochastic Optimization
- Variance Reduction for Decentralized Stochastic Optimization with Multiple Local Updates
- Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfe
- Optimal Momentum Methods for Stochastic Multilevel Compositional Optimization
- Vector- and operator-valued backward stochastic equations with finite-variation drivers and a maximum principle for singular stochastic control in infinite dimensions
BibTeX
@misc{everett2026dynamics,
title = {{Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions}},
author = {Katie Everett and Elliot Paquette},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.28961},
eprint = {2605.28961},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.28961},
}
RIS
TY - UNPB
TI - Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions
AU - Katie Everett
AU - Elliot Paquette
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2605.28961
UR - https://arxiv.org/abs/2605.28961
AB - Existing theory of momentum assumes that gradients arrive at every parameter at a roughly constant rate, an assumption violated in practice by heavy-tailed data distributions and modern architectures. We theoretically analyze the dynamics of two tractable models of momentum under sparse updates: a least squares model with sparse inputs and a logistic regression model with a rare class. Both admit exact closed-form second-moment dynamics whose high-dimensional limits we characterize across three scaling exponents for sparsity, batch size, and momentum decay. The phase structure on both problems is governed by the ratio of two intrinsic timescales: a momentum retention timescale (how many active updates the buffer survives) and a learning timescale (how many active updates it takes to reduce the squared error). When learning is much slower than retention, the limit matches SGD; when learning is faster, the system is unstable; where the timescales coincide, we recover classical heavy-ball dynamics. The oscillatory dynamics occur at different momentum values for different token sparsity, creating a spectral conflict for global momentum across token frequencies.
ER -