A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
Abstract
A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm updates induce an effective learning rate that grows as gradients shrink. We derive a dynamical mean-field theory (DMFT) describing the joint dependence of the loss on training time, model width and batch size. Normalization initially accelerates SGD, mapping the power-law exponent rSGD<1 to 2rSGD/(1−rSGD), with exponential convergence at rSGD=1 and formal finite-time convergence for rSGD>1. At finite step size, however, the same feedback ultimately breaks the acceleration and leads to marginal stability. The late-time theory yields width-limited, edge-of-stochastic-stability (EoSS), and deterministic edge-of-stability (EoS) regimes. These phases determine when larger batches or wider models reduce serial training time at comparable compute. We quantify in which of these phases increased batch size or width can compensate the excess compute use per step by fewer optimization steps to target loss. Linearized ResNet experiments on CIFAR-5M support the predicted acceleration, breakdown, and resource-scaling trends. Together, these results connect normalization-induced acceleration, EoS effects, and width--batch allocation within a solvable theory.
Related
- Restart and Adaptive Acceleration in Stochastic Gradient Methods
- On a blended Anderson / NGMRES acceleration for the Navier-Stokes Picard iteration: convergence analysis and adaptive depth selection
- Micro-to-macro derivation and stability analysis of an Aw-Rascle-Zhang-type model with driver-dependent acceleration
- Analysis of RCLUPPr: stability, robustness and acceleration
- In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
BibTeX
@misc{lavie2026solvable,
title = {{A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling}},
author = {Itay Lavie and Clarissa Lauditi and Cengiz Pehlevan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06701},
eprint = {2610.06701},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06701},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
AU - Itay Lavie
AU - Clarissa Lauditi
AU - Cengiz Pehlevan
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06701
UR - https://arxiv.org/abs/2610.06701
AB - A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm updates induce an effective learning rate that grows as gradients shrink. We derive a dynamical mean-field theory (DMFT) describing the joint dependence of the loss on training time, model width and batch size. Normalization initially accelerates SGD, mapping the power-law exponent r_SGD<1 to 2r_SGD/(1−r_SGD), with exponential convergence at r_SGD=1 and formal finite-time convergence for r_SGD>1. At finite step size, however, the same feedback ultimately breaks the acceleration and leads to marginal stability. The late-time theory yields width-limited, edge-of-stochastic-stability (EoSS), and deterministic edge-of-stability (EoS) regimes. These phases determine when larger batches or wider models reduce serial training time at comparable compute. We quantify in which of these phases increased batch size or width can compensate the excess compute use per step by fewer optimization steps to target loss. Linearized ResNet experiments on CIFAR-5M support the predicted acceleration, breakdown, and resource-scaling trends. Together, these results connect normalization-induced acceleration, EoS effects, and width--batch allocation within a solvable theory.
ER -