Early Memory Selection for Balanced Adam
Abstract
We propose a method for choosing the shared memory parameter β1=β2=β in Adam from a short pilot training. The selected β remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared β=0.95. The mean gap is also 32.3% lower than that of the best constant β chosen across all eleven workloads.
Related
- Early Selection of Polar Ensemble Decoding with Successive Cancellation Style Component Decoder
- Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
- Beyond Quadratic Loss: The Stability Phase Diagram of Adam
- Projection, Memory, and the Validity of Scalar Early-Warning Signals: A Schur--Volterra Theory with an AMOC Application
- From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
BibTeX
@misc{fernandezhernandez2026early,
title = {{Early Memory Selection for Balanced Adam}},
author = {Alberto Fernández-Hernández and Cristian Pérez-Corral and Jose I. Mestre and Manuel F. Dolz and Enrique S. Quintana-Ortí},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08624},
eprint = {2610.08624},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08624},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Early Memory Selection for Balanced Adam
AU - Alberto Fernández-Hernández
AU - Cristian Pérez-Corral
AU - Jose I. Mestre
AU - Manuel F. Dolz
AU - Enrique S. Quintana-Ortí
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08624
UR - https://arxiv.org/abs/2610.08624
AB - We propose a method for choosing the shared memory parameter β_1=β_2=β in Adam from a short pilot training. The selected β remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared β=0.95. The mean gap is also 32.3% lower than that of the best constant β chosen across all eleven workloads.
ER -