Variance Reduction Note 3: The Convergence of PAGE
Abstract
The probabilistic gradient estimator (PAGE) alternates randomly between an accurate gradient refresh and a cheap recursive correction. This third expository note on recursive variance reduction explains the algorithm and gives a self-contained convergence proof for smooth nonconvex optimization. A Lyapunov function combines objective decrease with estimation error, turning the refresh probability into an explicit contraction parameter. The resulting expected oracle complexity is O(n+√n ε−2) for finite sums and O(ε−3) for online problems, with the other problem parameters fixed. Initialization, refreshes, and both evaluations in each gradient difference are counted. The presentation is pedagogical and makes no claim of a new algorithm or convergence rate.
Related
- Variance Reduction Note 2: The Convergence of SPIDER
- Variance Reduction Note 4: The Convergence of SVRG
- A note on the convergence guarantees of RLT-based algorithms for polynomial optimization
- A note on the convergence analysis of Laguerre approximations for analytic functions
- Variance Reduction with Delayed Feedback for Non-Convex Optimization
BibTeX
@misc{lodhi2026variance,
title = {{Variance Reduction Note 3: The Convergence of PAGE}},
author = {Alina Lodhi},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01523},
eprint = {2611.01523},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01523},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Variance Reduction Note 3: The Convergence of PAGE
AU - Alina Lodhi
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01523
UR - https://arxiv.org/abs/2611.01523
AB - The probabilistic gradient estimator (PAGE) alternates randomly between an accurate gradient refresh and a cheap recursive correction. This third expository note on recursive variance reduction explains the algorithm and gives a self-contained convergence proof for smooth nonconvex optimization. A Lyapunov function combines objective decrease with estimation error, turning the refresh probability into an explicit contraction parameter. The resulting expected oracle complexity is O(n+√n ε^−2) for finite sums and O(ε^−3) for online problems, with the other problem parameters fixed. Initialization, refreshes, and both evaluations in each gradient difference are counted. The presentation is pedagogical and makes no claim of a new algorithm or convergence rate.
ER -