Kinetic theory for Transformers and the lost-in-the-middle phenomenon
Abstract
We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is U-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.
Related
- Convergence analysis of moment-based approximations in kinetic theory
- Fluctuating Kinetic Theory: A Poissonian Stochastic Boltzmann Equation
- A Kinetic Theory Approach To Ordered Fluids: Direct Simulation Monte Carlo Methods
- A Program for Fluctuating Kinetic Theory
- The Wave Kinetic Theory for Quasilinear MMT Equation
BibTeX
@misc{duerinckx2026kinetic,
title = {{Kinetic theory for Transformers and the lost-in-the-middle phenomenon}},
author = {Mitia Duerinckx and Borjan Geshkovski and Stefano Rossi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.09213},
eprint = {2605.09213},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.09213},
}
RIS
TY - UNPB
TI - Kinetic theory for Transformers and the lost-in-the-middle phenomenon
AU - Mitia Duerinckx
AU - Borjan Geshkovski
AU - Stefano Rossi
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2605.09213
UR - https://arxiv.org/abs/2605.09213
AB - We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is U-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.
ER -