Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
Abstract
We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general L-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by β1, and a preconditioner perturbation governed by β2. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit (β1,β2,ε)-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.
Related
- Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization
- Sparse Data Augmentation for Optimization with Provable Guarantees
- Why Forget-Only Unlearning Needs Memorization
- Why Constrained Updates Still Forget: A Functional-Path Analysis of Continual SFT
- A stationary harmonic map with a nonstationary tangent map
BibTeX
@misc{sahu2026adapt,
title = {{Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization}},
author = {Sharan Sahu and Abir Sarkar and Cameron J. Hogan and Martin T. Wells},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.04269},
eprint = {2605.04269},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.04269},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
AU - Sharan Sahu
AU - Abir Sarkar
AU - Cameron J. Hogan
AU - Martin T. Wells
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2605.04269
UR - https://arxiv.org/abs/2605.04269
AB - We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general L-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by β_1, and a preconditioner perturbation governed by β_2. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit (β_1,β_2,ε)-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.
ER -