Sharp Initialization Guarantees for the Heavy-Ball Method
Abstract
The heavy-ball method can accelerate convergence near the minimizer, but it may also enter persistent oscillations on general strongly convex objectives. How should it be initialized to reliably realize this local advantage? We study the initialization neighborhood in which convergence can be guaranteed using only curvature and regularity bounds. This neighborhood shrinks as momentum approaches one; for Lipschitz-continuous Hessians, its radius is of the same order as damping. Matching upper and lower bounds show that this initialization requirement cannot be relaxed uniformly over the function class. We then derive a startup rule based on the current gradient: a finite gradient-descent warm-up enters the guaranteed region before switching to heavy-ball. With parameters optimal for strongly convex quadratics, the subsequent iterates satisfy an accelerated local asymptotic rate bound. Code is available at https://anonymous.4open.science/r/HeavyBall-811D/.
Related
- A Lower Bound for the Heavy-Ball Method on Smooth Convex Functions
- Randomized average block coordinate descent method with heavy-ball momentum for tensor least squares problem under the t-product
- Heavy-Ball Method under Randomized Schedules
- Non-Acceleration of the Heavy-Ball Method in 1D
- Robust dimension-free estimation of simple random tensors: optimal guarantees under heavy tails and adversarial contamination
BibTeX
@misc{liu2026sharp,
title = {{Sharp Initialization Guarantees for the Heavy-Ball Method}},
author = {Kang Liu and Quanen Wu and Zhan Sun and Mengxiao Chen and Suyan Li and Bohao Qu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10369},
eprint = {2610.10369},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10369},
}
RIS
TY - UNPB
TI - Sharp Initialization Guarantees for the Heavy-Ball Method
AU - Kang Liu
AU - Quanen Wu
AU - Zhan Sun
AU - Mengxiao Chen
AU - Suyan Li
AU - Bohao Qu
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10369
UR - https://arxiv.org/abs/2610.10369
AB - The heavy-ball method can accelerate convergence near the minimizer, but it may also enter persistent oscillations on general strongly convex objectives. How should it be initialized to reliably realize this local advantage? We study the initialization neighborhood in which convergence can be guaranteed using only curvature and regularity bounds. This neighborhood shrinks as momentum approaches one; for Lipschitz-continuous Hessians, its radius is of the same order as damping. Matching upper and lower bounds show that this initialization requirement cannot be relaxed uniformly over the function class. We then derive a startup rule based on the current gradient: a finite gradient-descent warm-up enters the guaranteed region before switching to heavy-ball. With parameters optimal for strongly convex quadratics, the subsequent iterates satisfy an accelerated local asymptotic rate bound. Code is available at https://anonymous.4open.science/r/HeavyBall-811D/.
ER -