Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
Abstract
We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.
Related
- ProteinJEPA: Latent prediction improves protein language model pretraining
- Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining
- Effective Dynamics for the Bose Polaron in the Large-Volume Mean-Field Limit
- Effective threshold dynamics for the cubic nonlinear Klein-Gordon equation in three dimensions
- Effective Mapping Class Group Dynamics II: Geometric Intersection Numbers
BibTeX
@misc{liu2026effective,
title = {{Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining}},
author = {Zihan Liu and Ruiheng Zheng and Shaobo Zhang and Changxin Tian and Kunlong Chen and Zhiqiang Zhang and Lei Wu},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.24814},
eprint = {2608.24814},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.24814},
}
RIS
TY - UNPB
TI - Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
AU - Zihan Liu
AU - Ruiheng Zheng
AU - Shaobo Zhang
AU - Changxin Tian
AU - Kunlong Chen
AU - Zhiqiang Zhang
AU - Lei Wu
PY - 2026
DA - 2026/08/25
PB - arXiv
DO - 10.48550/arxiv.2608.24814
UR - https://arxiv.org/abs/2608.24814
AB - We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.
ER -