Second-Moment Stochastic Approximation Methods
Abstract
Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and second moments. In the second stage, we replace the exact moments with their respective estimators, and invoke Dvoretzky's theorem to show that the resulting practical methods converge almost surely to a neighborhood of the target solution. The size of the neighborhood depends on the biases and variances of the first- and second-moment estimators. We derive concrete bounds for Muon and a spectral variant of Adam that determine the radius of their neighborhood of convergence.
Related
- Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation
- Second moment of incomplete Eisenstein series revisited
- Moment Constants for Brownian Affine Approximation
- Extension on the convergence rates of moment-SOS hierarchies via approximation of truncated moment sequences
- Moment-optimal Gamma approximation of Bessel Process exit times
BibTeX
@misc{jiang2026secondmoment,
title = {{Second-Moment Stochastic Approximation Methods}},
author = {Tao Jiang and Lin Xiao},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.36600},
eprint = {2609.36600},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.36600},
}
RIS
TY - UNPB
TI - Second-Moment Stochastic Approximation Methods
AU - Tao Jiang
AU - Lin Xiao
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.36600
UR - https://arxiv.org/abs/2609.36600
AB - Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and second moments. In the second stage, we replace the exact moments with their respective estimators, and invoke Dvoretzky's theorem to show that the resulting practical methods converge almost surely to a neighborhood of the target solution. The size of the neighborhood depends on the biases and variances of the first- and second-moment estimators. We derive concrete bounds for Muon and a spectral variant of Adam that determine the radius of their neighborhood of convergence.
ER -