Skip to content
DraconiticResearch
Search Sources API
197,768 studies31 sourcesUpdated 55 min ago
PreprintarXiv licenceHAL

Restart and Adaptive Acceleration in Stochastic Gradient Methods

  • Ali Elhishi
  • Chistophe Roux
  • Alexandre d'Aspremont
Open at sourcePDFDOICite

Abstract

We study restart schemes in stochastic optimization problems for non-smooth and weakly convex functions that satisfy a Kurdyka-Łojasiewicz (KŁ) inequality. Using restarts allows us to leverage KŁ inequalities to achieve optimal convergence rates, with acceleration depending explicitly on the KŁ exponent. Furthermore, for stochastic gradient descent (SGD), optimal restart schedules correspond to piecewise constant (step decay) step sizes. While regularity constants such as the KŁ exponent are typically unknown in practice, we prove that restart schemes are robust to significant misspecification of these constants, hence nearly adaptive. We detail numerical experiments on a number of problems where the KŁ exponent is controlled.

Related

  1. A Riemannian Inertial Adaptive Gradient Method with Momentum Restart
    Lateef O. Jolaoso, Professer V. Ndlovu, Maggie Aphane
    arXivPreprintCC BY 4.0Full textOptimisation & controlarXiv
  2. Adaptive Restart under Unknown KL Geometry in Stochastic Weakly-Convex Optimization
    Binh T. Nguyen, Khoa D. Nguyen, Bao Thai Huynh
    arXivPreprintCC BY-NC-ND 4.0Full textOptimisation & controlarXiv
  3. Acceleration of Generalized Conditional Gradient Methods
    Long Chen, Zeyi Xu
    arXivPreprintarXiv licenceFull textOptimisation & controlarXiv
  4. Stochastic Gradient Methods with Online Scaling
    Wanyu Zhang, Wenzhi Gao, Yinyu Ye +1
    arXivPreprintCC BY 4.0Full textOptimisation & controlarXiv
  5. An Adaptive Projected-Gradient Algorithm for Sample-Average Approximations of Stochastic Multi-Objective Optimization
    Yiyang Li, Lei Wang, Xiaojun Chen
    arXivPreprintarXiv licenceFull textOptimisation & controlarXiv

Cite

BibTeXRIS
BibTeX
@misc{elhishi2026restart,
  title = {{Restart and Adaptive Acceleration in Stochastic Gradient Methods}},
  author = {Ali Elhishi and Chistophe Roux and Alexandre d'Aspremont},
  howpublished = {HAL},
  year = {2026},
  month = jun,
  doi = {10.48550/arxiv.2606.21354},
  eprint = {2606.21354},
  archivePrefix = {arXiv},
  url = {https://hal.science/hal-05761101v1},
}
RIS
TY  - UNPB
TI  - Restart and Adaptive Acceleration in Stochastic Gradient Methods
AU  - Ali Elhishi
AU  - Chistophe Roux
AU  - Alexandre d'Aspremont
PY  - 2026
DA  - 2026/06/19
PB  - HAL
DO  - 10.48550/arxiv.2606.21354
UR  - https://hal.science/hal-05761101v1
AB  - We study restart schemes in stochastic optimization problems for non-smooth and weakly convex functions that satisfy a Kurdyka-Łojasiewicz (KŁ) inequality. Using restarts allows us to leverage KŁ inequalities to achieve optimal convergence rates, with acceleration depending explicitly on the KŁ exponent. Furthermore, for stochastic gradient descent (SGD), optimal restart schedules correspond to piecewise constant (step decay) step sizes. While regularity constants such as the KŁ exponent are typically unknown in practice, we prove that restart schemes are robust to significant misspecification of these constants, hence nearly adaptive. We detail numerical experiments on a number of problems where the KŁ exponent is controlled.
ER  -