Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance
Abstract
Increasing the batch size during training is a common practice in large language model (LLM) pretraining, yet the theoretical justification behind its success is not well understood. Analyses of stochastic optimization often assume uniformly bounded stochastic gradient variance, yet recent evidence suggests that this assumption fails in many practical nonconvex problems. The Blum--Gladyshev (BG-$0$) noise model relaxes this assumption by allowing the variance to grow quadratically with the distance from initialization, suggesting that batch size schedulers can help by controlling the variance growth during training. However, this growth can be overly conservative in practice. We empirically investigate variance growth in LLM pretraining and observe that a generalized BG model with a tunable growth exponent provides a tighter description of practical noise behavior. Motivated by this observation, we introduce the generalized BG-a noise model, which interpolates between bounded variance (a=0) and BG-$0$ noise (a=2). Under L-smoothness, we derive an information-theoretic lower bound with growth-dependent oracle complexity Ω(ε−(4+a)) and establish a matching upper bound in ε-dependence by increasing the batch size as the iterates move away from initialization. Finally, we propose an adaptive batch scheduler that controls variance growth through dynamic batch size adjustments during training. In pretraining OLMo2 models of up to 1B parameters on C4, our scheduler achieves a lower validation loss than both small and large batch training under matched token budgets, while using less than 10\% of the iterations of small batch training.
Related
- Variance-adaptive Chernoff bounds: nonasymptotically valid, asymptotically sharp confidence sets for unbounded means
- A Localized Core-Tail Fourier-Laguerre Frame Method with Adaptive Frequency Detection for Unbounded-Domain Problems
- Adaptive Variance-Reduced Stochastic Sequential (Quadratic Constrained) Quadratic Programming
- VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion
- Adaptive Variance Reduction without Knowing Oracle Smoothness
BibTeX
@misc{fazla2026does,
title = {{Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance}},
author = {Arda Fazla and Antesh Upadhyay and Ege C. Kaya and M. Berk Sahin and Abolfazl Hashemi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02355},
eprint = {2610.02355},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02355},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance
AU - Arda Fazla
AU - Antesh Upadhyay
AU - Ege C. Kaya
AU - M. Berk Sahin
AU - Abolfazl Hashemi
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.02355
UR - https://arxiv.org/abs/2610.02355
AB - Increasing the batch size during training is a common practice in large language model (LLM) pretraining, yet the theoretical justification behind its success is not well understood. Analyses of stochastic optimization often assume uniformly bounded stochastic gradient variance, yet recent evidence suggests that this assumption fails in many practical nonconvex problems. The Blum--Gladyshev (BG-$0$) noise model relaxes this assumption by allowing the variance to grow quadratically with the distance from initialization, suggesting that batch size schedulers can help by controlling the variance growth during training. However, this growth can be overly conservative in practice. We empirically investigate variance growth in LLM pretraining and observe that a generalized BG model with a tunable growth exponent provides a tighter description of practical noise behavior. Motivated by this observation, we introduce the generalized BG-a noise model, which interpolates between bounded variance (a=0) and BG-$0$ noise (a=2). Under L-smoothness, we derive an information-theoretic lower bound with growth-dependent oracle complexity Ω(ε^−(4+a)) and establish a matching upper bound in ε-dependence by increasing the batch size as the iterates move away from initialization. Finally, we propose an adaptive batch scheduler that controls variance growth through dynamic batch size adjustments during training. In pretraining OLMo2 models of up to 1B parameters on C4, our scheduler achieves a lower validation loss than both small and large batch training under matched token budgets, while using less than 10\% of the iterations of small batch training.
ER -