Scaling Limits of Long-Context Transformers
Abstract
We study the long-context limit of softmax self-attention with a fixed query and a random context of n i.i.d. keys on the sphere with the intrinsic dimension of the key distribution held fixed, viewing the inverse temperature βn as the scaling parameter that decides whether attention degenerates into uniform averaging or collapses onto the single closest key. We show that the critical scale at which selectivity emerges is determined by the local exponent of the distance-to-query distribution near zero rather than by global features of the context, and scales like βn∗ asymp n2/(d−1) for uniform keys on 𝕊d−1. Furthermore, we characterize the limiting laws of the ordered attention weights and of the attention output across all regimes of βn: a subcritical regime in which the output reduces to a local average around q with explicit deterministic bias and Gaussian fluctuations; a critical regime in which a finite collection of nearest keys retains macroscopic mass without single-key collapse; and a supercritical regime in which all mass concentrates on the closest key. Of notable interest is the subcritical case with identity value matrix where the attention map approximately implements a backward heat equation.
Related
- Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers
- Diffusion Transformers are Provably Optimal In-context Generators
- Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
BibTeX
@misc{bruno2026scaling,
title = {{Scaling Limits of Long-Context Transformers}},
author = {Giuseppe Bruno and Shi Chen and Zhengjiang Lin and Yury Polyanskiy and Philippe Rigollet},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.08505},
eprint = {2605.08505},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.08505},
}
RIS
TY - UNPB
TI - Scaling Limits of Long-Context Transformers
AU - Giuseppe Bruno
AU - Shi Chen
AU - Zhengjiang Lin
AU - Yury Polyanskiy
AU - Philippe Rigollet
PY - 2026
DA - 2026/11/03
PB - arXiv
DO - 10.48550/arxiv.2605.08505
UR - https://arxiv.org/abs/2605.08505
AB - We study the long-context limit of softmax self-attention with a fixed query and a random context of n i.i.d. keys on the sphere with the intrinsic dimension of the key distribution held fixed, viewing the inverse temperature β_n as the scaling parameter that decides whether attention degenerates into uniform averaging or collapses onto the single closest key. We show that the critical scale at which selectivity emerges is determined by the local exponent of the distance-to-query distribution near zero rather than by global features of the context, and scales like β_n^∗ asymp n^2/(d−1) for uniform keys on 𝕊^d−1. Furthermore, we characterize the limiting laws of the ordered attention weights and of the attention output across all regimes of β_n: a subcritical regime in which the output reduces to a local average around q with explicit deterministic bias and Gaussian fluctuations; a critical regime in which a finite collection of nearest keys retains macroscopic mass without single-key collapse; and a supercritical regime in which all mass concentrates on the closest key. Of notable interest is the subcritical case with identity value matrix where the attention map approximately implements a backward heat equation.
ER -