Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems
Abstract
Last-iterate convergence guarantees for stochastic gradient methods have attracted significant attention as practical implementations typically return the last iterate instead of an average of the iterates. In this paper, we study two approaches for analyzing the last-iterate convergence of proximal and projected stochastic gradient methods without bounded domain or bounded variance assumptions, where black-box variational inference serves as a running example. Under two related assumptions relaxing the bounded variance, we show near-optimal anytime rate 𝒪̃(1/√T) for the last iterate. Moreover, when the horizon T is known, we remove the logarithmic factor and obtain the optimal order 𝒪(1/√T), matching the order known under bounded variance or bounded domain assumptions.
Related
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- On the convergence of doubly stochastic block gradient descent method for linear inverse problems
- On the Last-Iterate Convergence of Clipped Gradient Methods
- Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
BibTeX
@misc{vaidyan2026lastiterate,
title = {{Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems}},
author = {Kevin Kurian Thomas Vaidyan and Ahmet Alacaoglu and Michael P. Friedlander},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.07334},
eprint = {2611.07334},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07334},
}
RIS
TY - UNPB
TI - Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems
AU - Kevin Kurian Thomas Vaidyan
AU - Ahmet Alacaoglu
AU - Michael P. Friedlander
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2611.07334
UR - https://arxiv.org/abs/2611.07334
AB - Last-iterate convergence guarantees for stochastic gradient methods have attracted significant attention as practical implementations typically return the last iterate instead of an average of the iterates. In this paper, we study two approaches for analyzing the last-iterate convergence of proximal and projected stochastic gradient methods without bounded domain or bounded variance assumptions, where black-box variational inference serves as a running example. Under two related assumptions relaxing the bounded variance, we show near-optimal anytime rate 𝒪̃(1/√T) for the last iterate. Moreover, when the horizon T is known, we remove the logarithmic factor and obtain the optimal order 𝒪(1/√T), matching the order known under bounded variance or bounded domain assumptions.
ER -