Emergent One-Third Scaling Law as Attention Tries to Concentrate
Abstract
The neural scaling law relating longer training to better performance through a power law is central to today's large language models (LLMs), yet its origin remains debated. One recent proposal is that power laws can emerge from the strong non-linearity of a single softmax head learning peaked distributions. What happens with multiple softmax functions, as in LLMs, is unclear. Here, we show through toy models that any softmax learning peaked distributions, regardless of its position in the model, can develop logit magnitudes that grow in a power law with exponent 1/3, becoming a training bottleneck whose loss contribution decays as a power law with the same exponent 1/3. The overall loss therefore obeys 1/3 scaling whenever at least one softmax learns peaked distributions. We confirm that many softmax functions in LLMs learn peaked distributions and that LLM loss scaling matches this 1/3 prediction. Moreover, logit growth dynamics reveal that attention heads, rather than the language modeling head, are the bottleneck likely driving the 1/3 loss scaling in LLMs. Attention trying to concentrate on specific information, which is the heart of Transformers, may therefore also be the heart of the neural scaling law of training.
Related
- The Third law of thermodynamics for the Wheeler-DeWitt Crystal
- Dissipative Systems Analysis and Control. Theory and Applications: Erratum/Addendum. Springer Nature Switzerland AG, Communications and Control Engineering, Third uncorrected Edition, 2020
- A Math Circle in Third Grade
- Pfaffian Geometry and Indicial Singularities of Third-Order Linear ODEs
- The third symmetric product of a set is a set in HoTT
BibTeX
@misc{liu2026emergent,
title = {{Emergent One-Third Scaling Law as Attention Tries to Concentrate}},
author = {Yizhou Liu and Sara Kangaslahti and Jeff Gore},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32100},
eprint = {2609.32100},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32100},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Emergent One-Third Scaling Law as Attention Tries to Concentrate
AU - Yizhou Liu
AU - Sara Kangaslahti
AU - Jeff Gore
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2609.32100
UR - https://arxiv.org/abs/2609.32100
AB - The neural scaling law relating longer training to better performance through a power law is central to today's large language models (LLMs), yet its origin remains debated. One recent proposal is that power laws can emerge from the strong non-linearity of a single softmax head learning peaked distributions. What happens with multiple softmax functions, as in LLMs, is unclear. Here, we show through toy models that any softmax learning peaked distributions, regardless of its position in the model, can develop logit magnitudes that grow in a power law with exponent 1/3, becoming a training bottleneck whose loss contribution decays as a power law with the same exponent 1/3. The overall loss therefore obeys 1/3 scaling whenever at least one softmax learns peaked distributions. We confirm that many softmax functions in LLMs learn peaked distributions and that LLM loss scaling matches this 1/3 prediction. Moreover, logit growth dynamics reveal that attention heads, rather than the language modeling head, are the bottleneck likely driving the 1/3 loss scaling in LLMs. Attention trying to concentrate on specific information, which is the heart of Transformers, may therefore also be the heart of the neural scaling law of training.
ER -