Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule
Abstract
We study the implicit bias of Adam and its similarity to sign gradient descent on homogeneous models in binary classification tasks. Perhaps surprisingly, we uncover that these depend on the learning rate schedule. While previous works studied strongly decaying learning rate schedules η(t)=o (t1/L−1), where L is the homogeneity degree, and showed ℓ∞-margin-maximization biases, we focus on the more common complementary regime, including constant and weakly decaying schedules. For a polynomial schedule tα and homogeneity degree L, we show that when α> 1/L − 1/2, Adam has no trajectory-independent margin-maximization bias for deep linear networks (in the KKT sense): first, there is no non-trivial, locally Lipschitz, convex implicit regularizer in this setting, including any squared norm. Second, on width-$1$ networks, the result holds without requiring convexity, implying that any implicit regularizer for wide networks must be trivial when restricted to a width-$1$ network. Nevertheless, we positively characterize the implicit bias in this regime as maximizing a trajectory-dependent weighted-ℓ2 margin for any L-homogeneous model. The fundamental mechanism behind our results is convergence of Adam's preconditioner to a trajectory-dependent vector, which also allows showing that the limiting parameter direction depends on initialization. In the intermediate regime 1/L − 1 ≤ α≤ 1/L − 1/2 we prove an ℓ∞ bias, conditional on gradient directional convergence to a full-support vector. Of independent interest, we establish a loss decay rate for full batch Adam on general homogeneous models, whose leading order depends only on the second-moment parameter.
Related
- Adversarial Resilience of Poisson-Process Submodular Maximization over Matroids, and Full-Bandit Learning
- Optimal VC Dimension of Contrastive Learning with Margin
- Hierarchical Exponential-Gaussian Mixtures for Watch-Time Distribution Prediction
- Schedule optimization for tau-leaping in masked discrete diffusion
- Anytime Training with Schedule-Free Spectral Optimization
BibTeX
@misc{gronich2026watch,
title = {{Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule}},
author = {Eitan Gronich and Gal Vardi},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01929},
eprint = {2611.01929},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01929},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule
AU - Eitan Gronich
AU - Gal Vardi
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01929
UR - https://arxiv.org/abs/2611.01929
AB - We study the implicit bias of Adam and its similarity to sign gradient descent on homogeneous models in binary classification tasks. Perhaps surprisingly, we uncover that these depend on the learning rate schedule. While previous works studied strongly decaying learning rate schedules η(t)=o (t^1/L−1), where L is the homogeneity degree, and showed ℓ_∞-margin-maximization biases, we focus on the more common complementary regime, including constant and weakly decaying schedules. For a polynomial schedule t^α and homogeneity degree L, we show that when α> 1/L − 1/2, Adam has no trajectory-independent margin-maximization bias for deep linear networks (in the KKT sense): first, there is no non-trivial, locally Lipschitz, convex implicit regularizer in this setting, including any squared norm. Second, on width-$1$ networks, the result holds without requiring convexity, implying that any implicit regularizer for wide networks must be trivial when restricted to a width-$1$ network. Nevertheless, we positively characterize the implicit bias in this regime as maximizing a trajectory-dependent weighted-ℓ_2 margin for any L-homogeneous model. The fundamental mechanism behind our results is convergence of Adam's preconditioner to a trajectory-dependent vector, which also allows showing that the limiting parameter direction depends on initialization. In the intermediate regime 1/L − 1 ≤ α≤ 1/L − 1/2 we prove an ℓ_∞ bias, conditional on gradient directional convergence to a full-support vector. Of independent interest, we establish a loss decay rate for full batch Adam on general homogeneous models, whose leading order depends only on the second-moment parameter.
ER -