Skip to content
DraconiticResearch
Search Sources API
203,203 studies31 sourcesUpdated 7 min ago
PreprintCC BY 4.0arXiv

Asymptotic Behavior of Transformers

  • Katy Blumer
  • Da-Cheng Juan
  • Jon Kleinberg
  • Ravi Kumar
  • Cyrus Rashtchian
  • Andrew Tomkins
Open at sourcePDFDOICite

Abstract

In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.

Related

  1. Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
    Claudio Altafini
    arXivPreprintCC BY 4.0Full textDynamical systemsarXiv
  2. Serrin-type overdetermination in scaling limits: Sharp existence and asymptotic behavior of quasi-linear Schrödinger energy ground states
    Louis Jeanjean, Jianjun Zhang, Xuexiu Zhong
    Transactions of the American Mathematical SocietyPeer-reviewedLicence not statedFull text10 citedGeneral mathematicsOpenAlex
  3. A quasi-linear Schrödinger equation for nucleons : existence, uniqueness and asymptotic behavior
    Meriem Bahhi, Jonas Lampart, Simona Rota Nodari
    HALPreprintCC BY 4.0Full textAnalysis & PDEsHALarXiv
  4. Asymptotic delta-like behavior of the weighted taper in Thomson's multitaper F-test
    Hau-Tieng Wu
    arXivPreprintarXiv licenceFull textAnalysis & PDEsStatisticsarXiv
  5. Asymptotic behavior for the damped Schrödinger equation with nonlinear dissipation
    Takahisa Inui, Shun Takizawa
    arXivPreprintarXiv licenceFull textAnalysis & PDEsarXiv

Cite

BibTeXRIS
BibTeX
@misc{blumer2026asymptotic,
  title = {{Asymptotic Behavior of Transformers}},
  author = {Katy Blumer and Da-Cheng Juan and Jon Kleinberg and Ravi Kumar and Cyrus Rashtchian and Andrew Tomkins},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.19358},
  eprint = {2610.19358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.19358},
  note = {CC BY 4.0},
}
RIS
TY  - UNPB
TI  - Asymptotic Behavior of Transformers
AU  - Katy Blumer
AU  - Da-Cheng Juan
AU  - Jon Kleinberg
AU  - Ravi Kumar
AU  - Cyrus Rashtchian
AU  - Andrew Tomkins
PY  - 2026
DA  - 2026/10/14
PB  - arXiv
DO  - 10.48550/arxiv.2610.19358
UR  - https://arxiv.org/abs/2610.19358
AB  - In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.
ER  -