The Best Optimizer Depends on Batch Size
Abstract
A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size. Hyperparameter scaling rules promise to preserve performance as batch size and gradient noise change, suggesting that the best optimizer at one batch size should remain the best at another. We challenge this approach to developing and evaluating optimizers by showing: (1) no principled scaling rule for Muon works consistently across training settings, and (2) the best optimizer for language model pretraining changes with batch size even after extensive hyperparameter tuning. To understand these failures, we analyze a noisy quadratic model and show that the batch-size scaling rule that preserves performance is direction-dependent, shifting from square-root to linear as a direction's curvature-to-noise ratio rises. We further prove that gradient noise drives a bias-variance tradeoff in preconditioning that can reverse optimizer rankings across batch sizes, even under optimal learning rate and momentum tuning. Consistent with this, when we increase the batch size 16× partway through language model pretraining, keeping small-batch updates in only the sharpest 0.001% of directions removes up to 59.5% of the resulting loss gap over the next 8% of training, while a random subspace of the same size barely helps. Taken together, our results show that principled optimizer design must account for batch size and empirical benchmarking should be performed across batch sizes.
Related
- Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency
- Causal inference in staggered adoption panels: The correct comparison set depends on your intervention
- Multiplicative vs. Additive State Coupling in Learned Optimizer Schedules
- Nash-Bargaining HalpernSGD via Limited-Memory MultiLRSGA: A Two-Phase Optimizer for Multi-Objective Learning
- Explicit counterexamples to Frank-Loss optimizer conjectures for critical Dirac and vector Sobolev inequalities
BibTeX
@misc{dang2026best,
title = {{The Best Optimizer Depends on Batch Size}},
author = {Xingyu Dang and Kaiyue Wen and Sadhika Malladi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08975},
eprint = {2610.08975},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08975},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Best Optimizer Depends on Batch Size
AU - Xingyu Dang
AU - Kaiyue Wen
AU - Sadhika Malladi
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.08975
UR - https://arxiv.org/abs/2610.08975
AB - A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size. Hyperparameter scaling rules promise to preserve performance as batch size and gradient noise change, suggesting that the best optimizer at one batch size should remain the best at another. We challenge this approach to developing and evaluating optimizers by showing: (1) no principled scaling rule for Muon works consistently across training settings, and (2) the best optimizer for language model pretraining changes with batch size even after extensive hyperparameter tuning. To understand these failures, we analyze a noisy quadratic model and show that the batch-size scaling rule that preserves performance is direction-dependent, shifting from square-root to linear as a direction's curvature-to-noise ratio rises. We further prove that gradient noise drives a bias-variance tradeoff in preconditioning that can reverse optimizer rankings across batch sizes, even under optimal learning rate and momentum tuning. Consistent with this, when we increase the batch size 16× partway through language model pretraining, keeping small-batch updates in only the sharpest 0.001% of directions removes up to 59.5% of the resulting loss gap over the next 8% of training, while a random subspace of the same size barely helps. Taken together, our results show that principled optimizer design must account for batch size and empirical benchmarking should be performed across batch sizes.
ER -