Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise
Abstract
Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional p-th moments, p≥2. Our main result gives explicit accuracy and confidence conditions under which, for p>2, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At p=2, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.
Related
- Staircase phase transitions for the largest eigenvalue of heavy-tailed sample correlation matrices
- On the sample complexity of the generalized hidden shift problem over arbitrary finite groups
- Last-Iterate Convergence of Clipped SGD with Linear Decay
- Finite-Sample LTI System Identification from a Misspecified Finite Model Set
- Precise Convergence Speed of Clipped SGD
BibTeX
@misc{suvorikova2026clipped,
title = {{Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise}},
author = {Alexandra Suvorikova and Egor Gladin and Darina Dvinskikh and Artem Agafonov and Mohammad Alkousa and Yuriy Dorn and Vladislav Matyukhin and Alexander Gasnikov},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32962},
eprint = {2609.32962},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32962},
}
RIS
TY - UNPB
TI - Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise
AU - Alexandra Suvorikova
AU - Egor Gladin
AU - Darina Dvinskikh
AU - Artem Agafonov
AU - Mohammad Alkousa
AU - Yuriy Dorn
AU - Vladislav Matyukhin
AU - Alexander Gasnikov
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2609.32962
UR - https://arxiv.org/abs/2609.32962
AB - Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional p-th moments, p≥2. Our main result gives explicit accuracy and confidence conditions under which, for p>2, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At p=2, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.
ER -