Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference
Abstract
This paper develops strong Gaussian approximations (KMT type) for constant learning rate stochastic gradient descent (SGD). We work with heavy tailed noise and assume that the number of iterates grows and the learning rate shrinks simultaneously. Our work constructs simple Gaussian processes on the same space and bounds, in probability, the maximum difference between the SGD errors and their respective limits. While there exist central limit theorems for constant rate SGD centered at the optimum, these only provide individual iterate inference. By contrast, our quenched KMT approximations allow for finite sample, simultaneous, whole-trajectory-level inference and imply stochastic process convergence (Functional Central Limit Theorems (FCLTs)). For accumulated errors centered at the true optimum, we first obtain a KMT-type approximation bound when the learning rate scales as γn=c/√n+o(n−1/2) for a constant c ∈ ℝ+ and n iterations. The result reveals a trade off between controlling the bias of SGD and the initialization effect, with no learning rate scaling generally making both contributions vanish in the accumulated error limit. To eliminate these contributions, we consider a Burn in scheme and Richardson-Romberg extrapolation and prove respective KMT and FCLT theorems. We also prove a finite-horizon KMT-type approximation and FCLT for the rescaled iterate trajectory to a stationary Ornstein--Uhlenbeck (OU) process. We consider applications in risk control and simultaneous inference and perform simulations to illustrate our results.
Related
- Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD
- Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation
- Cubature-Based Statistical Moment Steering for Nonlinear, Non-Gaussian Trajectory Optimization
- Arithmetic purity of strong approximation for toric varieties with constant global sections
- Beyond Constant Truncations: The Bounded Sobolev Approximation Gap
BibTeX
@misc{keslin2026strong,
title = {{Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference}},
author = {Joseph Keslin and Wei Biao Wu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.14567},
eprint = {2610.14567},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.14567},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference
AU - Joseph Keslin
AU - Wei Biao Wu
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.14567
UR - https://arxiv.org/abs/2610.14567
AB - This paper develops strong Gaussian approximations (KMT type) for constant learning rate stochastic gradient descent (SGD). We work with heavy tailed noise and assume that the number of iterates grows and the learning rate shrinks simultaneously. Our work constructs simple Gaussian processes on the same space and bounds, in probability, the maximum difference between the SGD errors and their respective limits. While there exist central limit theorems for constant rate SGD centered at the optimum, these only provide individual iterate inference. By contrast, our quenched KMT approximations allow for finite sample, simultaneous, whole-trajectory-level inference and imply stochastic process convergence (Functional Central Limit Theorems (FCLTs)). For accumulated errors centered at the true optimum, we first obtain a KMT-type approximation bound when the learning rate scales as γ_n=c/√n+o(n^−1/2) for a constant c ∈ ℝ^+ and n iterations. The result reveals a trade off between controlling the bias of SGD and the initialization effect, with no learning rate scaling generally making both contributions vanish in the accumulated error limit. To eliminate these contributions, we consider a Burn in scheme and Richardson-Romberg extrapolation and prove respective KMT and FCLT theorems. We also prove a finite-horizon KMT-type approximation and FCLT for the rescaled iterate trajectory to a stationary Ornstein--Uhlenbeck (OU) process. We consider applications in risk control and simultaneous inference and perform simulations to illustrate our results.
ER -