Skip to content
DraconiticResearch
Search Sources API
190,851 studies31 sourcesUpdated 1 min ago
PreprintCC BY 4.0arXiv

Stochastic Gradient Methods with Online Scaling

  • Wanyu Zhang
  • Wenzhi Gao
  • Yinyu Ye
  • Madeleine Udell
Open at sourcePDFDOICite

Abstract

This paper introduces Stochastic Online Scaled Gradient Methods (SOSGM), a generalization of the recently developed adaptive preconditioning framework in arXiv:2505.23081 and arXiv:2509.11007 to stochastic optimization. Under standard assumptions, we establish convergence guarantees for SOSGM using large batchsize or variance reduction. SOSGM is compatible with popular diagonal and/or low-rank preconditioners as well as heavy-ball momentum, while maintaining memory and computation cost comparable to Adam. Extensive numerical experiments demonstrate the strong empirical performance of SOSGM. Using a diagonal preconditioner, SOSGM and its variants substantially outperform existing adaptive first-order methods across a range of statistical learning tasks.

Related

  1. Operator Splitting Methods with Online Scaling
    Wanyu Zhang, Wenzhi Gao, Madeleine Udell
    arXivPreprintCC BY 4.0Full textOptimisation & controlarXiv
  2. Online Covariance Estimation for Averaged Stochastic Gradient Descent on Riemannian Manifolds
    Jingrong Guan, Liwei Jiang, Sen Na
    arXivPreprintCC BY 4.0Full textOptimisation & controlStatisticsarXiv
  3. Resource-Adaptive Stochastic Gradient Descent for Online Linear Programming without Re-solving
    Jiameng Lyu
    arXivPreprintarXiv licenceFull textOptimisation & controlarXiv
  4. Restart and Adaptive Acceleration in Stochastic Gradient Methods
    Ali Elhishi, Chistophe Roux, Alexandre d'Aspremont
    HALPreprintarXiv licenceFull textOptimisation & controlHALarXiv
  5. Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems
    Kevin Kurian Thomas Vaidyan, Ahmet Alacaoglu, Michael P. Friedlander
    arXivPreprintarXiv licenceFull textOptimisation & controlStatisticsarXiv

Cite

BibTeXRIS
BibTeX
@misc{zhang2026stochastic,
  title = {{Stochastic Gradient Methods with Online Scaling}},
  author = {Wanyu Zhang and Wenzhi Gao and Yinyu Ye and Madeleine Udell},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.11751},
  eprint = {2609.11751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.11751},
  note = {CC BY 4.0},
}
RIS
TY  - UNPB
TI  - Stochastic Gradient Methods with Online Scaling
AU  - Wanyu Zhang
AU  - Wenzhi Gao
AU  - Yinyu Ye
AU  - Madeleine Udell
PY  - 2026
DA  - 2026/09/11
PB  - arXiv
DO  - 10.48550/arxiv.2609.11751
UR  - https://arxiv.org/abs/2609.11751
AB  - This paper introduces Stochastic Online Scaled Gradient Methods (SOSGM), a generalization of the recently developed adaptive preconditioning framework in arXiv:2505.23081 and arXiv:2509.11007 to stochastic optimization. Under standard assumptions, we establish convergence guarantees for SOSGM using large batchsize or variance reduction. SOSGM is compatible with popular diagonal and/or low-rank preconditioners as well as heavy-ball momentum, while maintaining memory and computation cost comparable to Adam. Extensive numerical experiments demonstrate the strong empirical performance of SOSGM. Using a diagonal preconditioner, SOSGM and its variants substantially outperform existing adaptive first-order methods across a range of statistical learning tasks.
ER  -