Beyond Quadratic Loss: The Stability Phase Diagram of Adam
Abstract
Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the (β1,β2) plane. Across a range of model--task settings, an approximately linear boundary, 1−β2=C(1−β1), separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss L(x)∝|x|n recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent n. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.
Related
- Rényi stability of Bh sets: a two-order phase diagram and sharp deletion principles
- Three thresholds and a computer-assisted phase diagram for the spectral stability of periodic hydroelastic Whitham waves
- Task-Oriented Quantization for Quadratic Scheduling: Centroid Water-Filling and Power-Diagram Encoders
- Funk Beta Balls: Exact Potentials and a Three-Phase Pyramid Diagram
- Phase diagram and fluctuations in the cubic mean field Ising model
BibTeX
@misc{tang2026beyond,
title = {{Beyond Quadratic Loss: The Stability Phase Diagram of Adam}},
author = {Gaoxiang Tang and Huanran Chen and Ziming Liu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.18314},
eprint = {2609.18314},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.18314},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Beyond Quadratic Loss: The Stability Phase Diagram of Adam
AU - Gaoxiang Tang
AU - Huanran Chen
AU - Ziming Liu
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.18314
UR - https://arxiv.org/abs/2609.18314
AB - Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the (β_1,β_2) plane. Across a range of model--task settings, an approximately linear boundary, 1−β_2=C(1−β_1), separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss L(x)∝|x|^n recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent n. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.
ER -