Variance Reduction Note 2: The Convergence of SPIDER
Abstract
The stochastic path-integrated differential estimator (SPIDER) tracks a gradient by accumulating small, shared-sample gradient differences and periodically refreshing the estimate. This second expository note on recursive variance reduction explains SPIDER-SFO and gives a self-contained proof of its expected first-order stationarity guarantee. The argument combines an epochwise mean-square error bound with descent under a clipped, normalized update. It yields O(ε−3) stochastic gradient evaluations in the online setting and O(n+√n ε−2) component gradient evaluations for a finite sum, with the remaining problem parameters fixed. All refresh costs are included. The presentation is pedagogical and makes no claim of a new algorithm or convergence rate.
Related
- Variance Reduction Note 3: The Convergence of PAGE
- Variance Reduction Note 4: The Convergence of SVRG
- A note on the convergence guarantees of RLT-based algorithms for polynomial optimization
- A note on the convergence analysis of Laguerre approximations for analytic functions
- Variance Reduction with Delayed Feedback for Non-Convex Optimization
BibTeX
@misc{lodhi2026variance,
title = {{Variance Reduction Note 2: The Convergence of SPIDER}},
author = {Alina Lodhi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.25304},
eprint = {2610.25304},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.25304},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Variance Reduction Note 2: The Convergence of SPIDER
AU - Alina Lodhi
PY - 2026
DA - 2026/10/18
PB - arXiv
DO - 10.48550/arxiv.2610.25304
UR - https://arxiv.org/abs/2610.25304
AB - The stochastic path-integrated differential estimator (SPIDER) tracks a gradient by accumulating small, shared-sample gradient differences and periodically refreshing the estimate. This second expository note on recursive variance reduction explains SPIDER-SFO and gives a self-contained proof of its expected first-order stationarity guarantee. The argument combines an epochwise mean-square error bound with descent under a clipped, normalized update. It yields O(ε^−3) stochastic gradient evaluations in the online setting and O(n+√n ε^−2) component gradient evaluations for a finite sum, with the remaining problem parameters fixed. All refresh costs are included. The presentation is pedagogical and makes no claim of a new algorithm or convergence rate.
ER -