Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
Abstract
The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the *active curvature*, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts *gradient reversal* of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.
Related
- Quasi-Static Fault-Tolerant Feedback Control of a Quadrotor under Rotor Failure with Provable Safety Guarantees
- Adaptive Local Proper Orthogonal Decomposition via Graph Partitioning and Provable Error-Bounded Lossy Compression for Parametric Series in Finite-Dimensional Function Spaces
- Oscillation of Second-Order Delay Equations with Positive Feedback
- Parallel Policy-Gradient Methods for Parameter Optimization of Nonlinear Feedback Controllers
- On hyperbolic PDEs, filtered feedback control laws, and fractal-like stability crossing curves
BibTeX
@misc{fong2026adam,
title = {{Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal}},
author = {Yiman Fong and Heng Yang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.20638},
eprint = {2608.20638},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.20638},
}
RIS
TY - UNPB
TI - Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
AU - Yiman Fong
AU - Heng Yang
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2608.20638
UR - https://arxiv.org/abs/2608.20638
AB - The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the *active curvature*, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts *gradient reversal* of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.
ER -