{
  "abstract": "We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general $L$-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by $β_1$, and a preconditioner perturbation governed by $β_2$. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit $(β_1,β_2,ε)$-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2605.04269",
  "author_total": 4,
  "authors": [
    {
      "name": "Sharan Sahu"
    },
    {
      "name": "Abir Sarkar"
    },
    {
      "name": "Cameron J. Hogan"
    },
    {
      "name": "Martin T. Wells"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/28vwpg71p5rtp.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2605.04269",
  "fulltext_url": null,
  "id": "28vwpg71p5rtp",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2605.04269",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "http://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "stat.ML"
    ],
    [
      "openalex",
      "1702|Stochastic Gradient Optimization Techniques"
    ]
  ],
  "openalex": "W7160490069",
  "pdf_url": "https://arxiv.org/pdf/2605.04269",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-11",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/28vwpg71p5rtp.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:21:17Z",
      "native_id": "2605.04269",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2605.04269"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ML",
      "subfield": "statistics"
    }
  ],
  "title": "Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization",
  "title_text": "Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization",
  "type": "preprint",
  "updated_at": "2026-10-30T12:21:17Z",
  "url": "https://research.draconitic.com/paper/28vwpg71p5rtp",
  "venue": "arXiv",
  "versions": []
}