Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
Abstract
Stochastic gradient descent (SGD) with gradient clipping and additive noise has become a standard technique for training machine learning models, particularly in applications requiring robustness or privacy guarantees. However, clipping introduces a bias in stochastic gradients, while additive noise introduces additional variance, making the long-run behaviour of individual optimization trajectories difficult to characterize. In this work, we prove that SGD with clipping and additive Gaussian noise (SGD-CN) converges almost surely (a.s.) under smoothness and uniformly bounded stochastic-gradient noise assumptions, provided the step sizes satisfy some standard decaying conditions. Our analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, where we show that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods and suggest that, despite the bias and noise introduced by clipping and perturbation, the algorithm remains stable in both convex and nonconvex regimes.
Related
- Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and Hölder Smoothness
- Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
- Convergence of finite element approximations for the one-dimensional stochastic Burgers equation with additive trace-class noise
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
- On The Linear Convergence of Bregman Proximal Gradient Methods with Applications to Kullback--Leibler regression
BibTeX
@misc{mukherjee2026almost,
title = {{Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise}},
author = {Amartya Mukherjee and Jun Liu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12119},
eprint = {2609.12119},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12119},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
AU - Amartya Mukherjee
AU - Jun Liu
PY - 2026
DA - 2026/09/16
PB - arXiv
DO - 10.48550/arxiv.2609.12119
UR - https://arxiv.org/abs/2609.12119
AB - Stochastic gradient descent (SGD) with gradient clipping and additive noise has become a standard technique for training machine learning models, particularly in applications requiring robustness or privacy guarantees. However, clipping introduces a bias in stochastic gradients, while additive noise introduces additional variance, making the long-run behaviour of individual optimization trajectories difficult to characterize. In this work, we prove that SGD with clipping and additive Gaussian noise (SGD-CN) converges almost surely (a.s.) under smoothness and uniformly bounded stochastic-gradient noise assumptions, provided the step sizes satisfy some standard decaying conditions. Our analysis extends to momentum variants such as the stochastic heavy ball and Nesterov's accelerated gradient, where we show that careful energy constructions yield similar guarantees. These results provide stronger theoretical foundations for understanding the pathwise behaviour of clipped stochastic gradient methods and suggest that, despite the bias and noise introduced by clipping and perturbation, the algorithm remains stable in both convex and nonconvex regimes.
ER -