Skip to content
DraconiticResearch
Search Sources API
203,203 studies31 sourcesUpdated 47 min ago
PreprintarXiv licencearXiv

A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

  • Younghun Lee
  • Jake Zummo
  • Wei Biao Wu
Open at sourcePDFDOICite

Abstract

Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite pth moment for p > 1, yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.

Related

  1. The Bias of Nonlinear Two-Time-scale Stochastic Approximation under Constant Step-Sizes
    Djamel Rassem Lamouri, Dorian Baudry, Nicolas Gast
    arXivPreprintCC BY 4.0Full textOptimisation & controlStatisticsarXiv
  2. Function-Valued Causal Influence in Nonlinear Time Series
    Valentina V. Kuskova, Dmitry Zaytsev, Michael Coppedge
    arXivPreprintCC BY 4.0Full textStatisticsarXiv
  3. Bayesian Quantile Deep Echo State Networks for Nonlinear Time Series
    Antonio De Leon, Raquel Prado, Bruno Sansó
    arXivPreprintCC BY 4.0Full textStatisticsarXiv
  4. Stochastic Gradient Descent: A Nonlinear Time Series Perspective
    Jiaqi Li, Zhipeng Lou, Stefan Richter +1
    arXivPreprintarXiv licenceFull textStatisticsarXiv
  5. Uniform Gaussian Approximation for The Quasi-Likelihood Estimator for a Weakly Dependent Nonlinear Time Series Models
    Zinsou Max Debaly, Arsene Brice Zotsa-Ngoufack
    arXivPreprintarXiv licenceFull textStatisticsarXiv

Cite

BibTeXRIS
BibTeX
@misc{lee2026nonlinear,
  title = {{A Nonlinear Time-Series Theory for Constant-step SGD with Momentum}},
  author = {Younghun Lee and Jake Zummo and Wei Biao Wu},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.16034},
  eprint = {2610.16034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.16034},
}
RIS
TY  - UNPB
TI  - A Nonlinear Time-Series Theory for Constant-step SGD with Momentum
AU  - Younghun Lee
AU  - Jake Zummo
AU  - Wei Biao Wu
PY  - 2026
DA  - 2026/10/12
PB  - arXiv
DO  - 10.48550/arxiv.2610.16034
UR  - https://arxiv.org/abs/2610.16034
AB  - Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite pth moment for p > 1, yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.
ER  -