Long-Time Behavior of Zero-Noise Mean-Field Adam
Abstract
Adaptive methods such as Adam are the most commonly used methods for training neural networks, but their long-time behavior is not as well understood as that of non-adaptive methods. This is particularly true in the mean-field limit, where the dynamics of infinitely many neurons are described as a distribution over parameters which satisfies a self-consistency condition. In this paper, we study a deterministic continuous-time model of Adam in the mean-field setting. Under a condition relating the averaging rates of the momentum and second-moment variables, the size of the gradients, and a regularization parameter, we prove that the distribution of parameters converges to the set of equilibria. This condition extends the range known for finite-dimensional models, and we show that it is sharp. Under an additional Lojasiewicz-type inequality for distributions, we prove that individual parameter trajectories converge and obtain quantitative rates for this convergence. We verify the qualitative convergence assumptions for a regularized two-layer neural network. We also show that adding noise to the momentum prevents the system from approaching these equilibria and rules out many free-energy type Lyapunov functionals. Finally, numerical experiments illustrate the energy estimates, give evidence of persistent oscillations outside the sufficient parameter range, and compare the observed decay with the predicted rates.
Related
- Long-Time Behavior of Mean Field Games on ℝd with Common White Noise and Quadratic Growth
- On the zero-noise limit for SDE’s singular at the initial time
- An Õ(n2 )-Time Sampler for Zero-Field Ferromagnetic Ising Models
- Zero temperature versus long time for diffusions driven by metric projections onto convex sets
- Long-time behavior of first-order Hamilton-Jacobi equations in the Wasserstein space
BibTeX
@misc{huffman2026longtime,
title = {{Long-Time Behavior of Zero-Noise Mean-Field Adam}},
author = {Ethan Huffman and Asaf Cohen},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01658},
eprint = {2611.01658},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01658},
}
RIS
TY - UNPB
TI - Long-Time Behavior of Zero-Noise Mean-Field Adam
AU - Ethan Huffman
AU - Asaf Cohen
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01658
UR - https://arxiv.org/abs/2611.01658
AB - Adaptive methods such as Adam are the most commonly used methods for training neural networks, but their long-time behavior is not as well understood as that of non-adaptive methods. This is particularly true in the mean-field limit, where the dynamics of infinitely many neurons are described as a distribution over parameters which satisfies a self-consistency condition. In this paper, we study a deterministic continuous-time model of Adam in the mean-field setting. Under a condition relating the averaging rates of the momentum and second-moment variables, the size of the gradients, and a regularization parameter, we prove that the distribution of parameters converges to the set of equilibria. This condition extends the range known for finite-dimensional models, and we show that it is sharp. Under an additional Lojasiewicz-type inequality for distributions, we prove that individual parameter trajectories converge and obtain quantitative rates for this convergence. We verify the qualitative convergence assumptions for a regularized two-layer neural network. We also show that adding noise to the momentum prevents the system from approaching these equilibria and rules out many free-energy type Lyapunov functionals. Finally, numerical experiments illustrate the energy estimates, give evidence of persistent oscillations outside the sufficient parameter range, and compare the observed decay with the predicted rates.
ER -