On the Last-Iterate Convergence of Clipped Gradient Methods
Abstract
In stochastic (strongly) convex optimization, establishing last-iterate convergence guarantees for first-order methods has recently garnered increasing attention, as this not only deepens the theoretical understanding of these algorithms but also aligns more closely with practice. Notably, people have demonstrated that one of the most famous, simple, and popular methods, Stochastic Gradient Descent (SGD), provably achieves last-iterate convergence. However, the existing results may have limited applicability, since they typically rely on strong conditions on the gradient noise (e.g., an exponentially decaying tail) rather than the more realistic heavy-tailed noise. Unfortunately, when faced with heavy-tailed noise, SGD is known to exhibit undesirable behavior or even fail to converge. To address the challenge of heavy-tailed noise, people have proposed Clipped−SGD, an algorithm that combines SGD with a simple mechanism, gradient clipping. Although Clipped−SGD performs well in practice, its last-iterate convergence property remains highly underexplored. In this work, we prove the first optimal last-iterate rates in high probability for Clipped−SGD under heavy-tailed noise, thereby closing a gap in the literature.
Related
- Last-Iterate Convergence of Clipped SGD with Linear Decay
- Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness
- Last-Iterate Convergence of Frank--Wolfe for Nonlinear Variational Inequalities on Polytopes
BibTeX
@misc{liu2026lastiterate,
title = {{On the Last-Iterate Convergence of Clipped Gradient Methods}},
author = {Zijian Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21026},
eprint = {2610.21026},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21026},
}
RIS
TY - UNPB
TI - On the Last-Iterate Convergence of Clipped Gradient Methods
AU - Zijian Liu
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.21026
UR - https://arxiv.org/abs/2610.21026
AB - In stochastic (strongly) convex optimization, establishing last-iterate convergence guarantees for first-order methods has recently garnered increasing attention, as this not only deepens the theoretical understanding of these algorithms but also aligns more closely with practice. Notably, people have demonstrated that one of the most famous, simple, and popular methods, Stochastic Gradient Descent (SGD), provably achieves last-iterate convergence. However, the existing results may have limited applicability, since they typically rely on strong conditions on the gradient noise (e.g., an exponentially decaying tail) rather than the more realistic heavy-tailed noise. Unfortunately, when faced with heavy-tailed noise, SGD is known to exhibit undesirable behavior or even fail to converge. To address the challenge of heavy-tailed noise, people have proposed Clipped−SGD, an algorithm that combines SGD with a simple mechanism, gradient clipping. Although Clipped−SGD performs well in practice, its last-iterate convergence property remains highly underexplored. In this work, we prove the first optimal last-iterate rates in high probability for Clipped−SGD under heavy-tailed noise, thereby closing a gap in the literature.
ER -