FlexGrad: a root-free approach to adaptive step sizes
Abstract
We introduce the FlexGrad optimizer, an adaptive gradient method that addresses a key limitation of classical AdaGrad-type methods: their inability to exploit correlations between gradients. By scaling steps using both cumulative gradients and cumulative squared gradients, FlexGrad preserves AdaGrad-like behavior in noisy regimes while allowing substantially larger steps when gradient directions are persistent. The method is fully online, horizon-free, and avoids any auxiliary search procedure during optimization. For convex Lipschitz objectives, we prove that FlexGrad attains the optimal asymptotic convergence rate for nonsmooth convex optimization, with guarantees for both global and coordinate-wise variants. FlexGrad is straightforward to implement and is closely connected to practical gradient-normalized methods. Experiments on a range of convex and nonconvex problems show that FlexGrad performs competitively across a range of optimization settings.
Related
- Locally adaptive step sizes for Hamiltonian Monte Carlo methods
- GRAAL for variational inequalities revisited: sharp parameter range and adaptive step sizes
- Cramér-type moderate deviations and precise large deviations for the elephant random walk with random step sizes
- Normal approximation, moderate and large deviations for the elephant random walk with stops and random step sizes
- Amnesic Elephant Random Walk with Polynomially Decaying Step Sizes
BibTeX
@misc{barros2026flexgrad,
title = {{FlexGrad: a root-free approach to adaptive step sizes}},
author = {Pablo Barros and Aaron Defazio and Vincent Guigues},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.09203},
eprint = {2610.09203},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.09203},
}
RIS
TY - UNPB
TI - FlexGrad: a root-free approach to adaptive step sizes
AU - Pablo Barros
AU - Aaron Defazio
AU - Vincent Guigues
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.09203
UR - https://arxiv.org/abs/2610.09203
AB - We introduce the FlexGrad optimizer, an adaptive gradient method that addresses a key limitation of classical AdaGrad-type methods: their inability to exploit correlations between gradients. By scaling steps using both cumulative gradients and cumulative squared gradients, FlexGrad preserves AdaGrad-like behavior in noisy regimes while allowing substantially larger steps when gradient directions are persistent. The method is fully online, horizon-free, and avoids any auxiliary search procedure during optimization. For convex Lipschitz objectives, we prove that FlexGrad attains the optimal asymptotic convergence rate for nonsmooth convex optimization, with guarantees for both global and coordinate-wise variants. FlexGrad is straightforward to implement and is closely connected to practical gradient-normalized methods. Experiments on a range of convex and nonconvex problems show that FlexGrad performs competitively across a range of optimization settings.
ER -