Variance Reduction Note 4: The Convergence of SVRG
Abstract
Stochastic variance-reduced gradient (SVRG) uses a full gradient at a snapshot point to correct inexpensive component gradients. This fourth expository note on variance reduction explains that control variate and gives a self-contained proof of linear convergence for the classical strongly convex finite-sum setting. Each component is assumed convex and smooth, while only their average must be strongly convex. A second-moment bound links the estimator to the objective gaps at the current iterate and the snapshot. Summing a distance recursion over an epoch then yields a geometric decrease and an oracle complexity of order (n+L/μ)log(1/τ) for objective accuracy τ, with the initial gap fixed. The result uses a random snapshot update and counts all full-gradient computations. No new algorithm or rate is claimed.
Related
- Variance Reduction Note 3: The Convergence of PAGE
- Variance Reduction Note 2: The Convergence of SPIDER
- A note on the convergence guarantees of RLT-based algorithms for polynomial optimization
- A note on the convergence analysis of Laguerre approximations for analytic functions
- Variance Reduction with Delayed Feedback for Non-Convex Optimization
BibTeX
@misc{lodhi2026variance,
title = {{Variance Reduction Note 4: The Convergence of SVRG}},
author = {Alina Lodhi},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.05075},
eprint = {2611.05075},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.05075},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Variance Reduction Note 4: The Convergence of SVRG
AU - Alina Lodhi
PY - 2026
DA - 2026/11/03
PB - arXiv
DO - 10.48550/arxiv.2611.05075
UR - https://arxiv.org/abs/2611.05075
AB - Stochastic variance-reduced gradient (SVRG) uses a full gradient at a snapshot point to correct inexpensive component gradients. This fourth expository note on variance reduction explains that control variate and gives a self-contained proof of linear convergence for the classical strongly convex finite-sum setting. Each component is assumed convex and smooth, while only their average must be strongly convex. A second-moment bound links the estimator to the objective gaps at the current iterate and the snapshot. Summing a distance recursion over an epoch then yields a geometric decrease and an oracle complexity of order (n+L/μ)log(1/τ) for objective accuracy τ, with the initial gap fixed. The result uses a random snapshot update and counts all full-gradient computations. No new algorithm or rate is claimed.
ER -