Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness
Abstract
Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practical implementations typically return the last iterate. In this paper, we study the last-iterate convergence of normalized gradient descent for convex, (ν,Mν)-Hölder-smooth objectives. For a constant stepsize, we establish an upper bound of 𝒪((log2(T)/T)(1+ν)/2), which contains a logarithmic overhead relative to the known 𝒪(T−(1+ν)/2) guarantees for the best and weighted-average iterates. For ν= 0, this overhead is known to be unavoidable. We complement this analysis with numerical results based on the performance estimation problem (PEP), investigating the finite-horizon worst-case behavior in the smooth setting and whether the logarithmic overhead reflects an intrinsic limitation of constant-step normalized gradient descent. We then show that a linearly decreasing stepsize yields a last-iterate guarantee of 𝒪(T−(1+ν)/2), matching the order of the best-iterate/weighted-average guarantees without requiring knowledge of ν and Mν.
Related
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- Last-Iterate Performance of Gradient Descent and Relaxed Proximal Point via s-Composability
- On the Last-Iterate Convergence of Clipped Gradient Methods
- Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and Hölder Smoothness
- PL conditions do not guarantee convergence of gradient descent-ascent dynamics
BibTeX
@misc{takezawa2026lastiterate,
title = {{Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness}},
author = {Yuki Takezawa and Eduard Gorbunov},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06070},
eprint = {2610.06070},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06070},
}
RIS
TY - UNPB
TI - Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness
AU - Yuki Takezawa
AU - Eduard Gorbunov
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06070
UR - https://arxiv.org/abs/2610.06070
AB - Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practical implementations typically return the last iterate. In this paper, we study the last-iterate convergence of normalized gradient descent for convex, (ν,M_ν)-Hölder-smooth objectives. For a constant stepsize, we establish an upper bound of 𝒪((log^2(T)/T)^(1+ν)/2), which contains a logarithmic overhead relative to the known 𝒪(T^−(1+ν)/2) guarantees for the best and weighted-average iterates. For ν= 0, this overhead is known to be unavoidable. We complement this analysis with numerical results based on the performance estimation problem (PEP), investigating the finite-horizon worst-case behavior in the smooth setting and whether the logarithmic overhead reflects an intrinsic limitation of constant-step normalized gradient descent. We then show that a linearly decreasing stepsize yields a last-iterate guarantee of 𝒪(T^−(1+ν)/2), matching the order of the best-iterate/weighted-average guarantees without requiring knowledge of ν and M_ν.
ER -