Gaussian Limits for SGD Without Stationary Moments
Abstract
Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with o(a) error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.
Related
- Stationary Limits of the Point Spectrum
- Subcritical limits of a 1D KPZ growth surface with finite moments
- Hamiltonian stationary Lagrangian surfaces of constant Gaussian curvature in complex space forms
- Correlated Gaussian Moments Determine Polynomials up to Orthogonal Equivalence
- Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD
BibTeX
@misc{li2026gaussian,
title = {{Gaussian Limits for SGD Without Stationary Moments}},
author = {Xiaoli Li and Wei Biao Wu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05599},
eprint = {2610.05599},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05599},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Gaussian Limits for SGD Without Stationary Moments
AU - Xiaoli Li
AU - Wei Biao Wu
PY - 2026
DA - 2026/10/04
PB - arXiv
DO - 10.48550/arxiv.2610.05599
UR - https://arxiv.org/abs/2610.05599
AB - Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with o(a) error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.
ER -