A Unified Perspective on the Dynamics of Deep Transformers
Abstract
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the iterative application of attention across layers induces complex dynamics that remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. Our first set of contributions focuses on compactly supported initial data. We show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system, thus generalizing and extending previous analysis to several variants of self-attention: multi-head attention, L2 attention, Sinkhorn attention, Sigmoid attention, and masked attention--leveraging a conditional Wasserstein framework. In a second set of contributions, we are the first to study non-compactly supported initial conditions, by focusing on Gaussian initial data. Again for different types of attention, we show that the Transformer PDE preserves the space of Gaussian measures, which allows us to analyze the Gaussian case theoretically and numerically to identify typical behaviors. This Gaussian analysis captures the evolution of data anisotropy through a deep Transformer. In particular, we highlight a clustering phenomenon that parallels previous results in the non-normalized discrete case.
Related
- When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers
- From Specific Heat Bounds to Penrose-like Inequalities: A Unified Thermodynamic Perspective
- Fermion−Monopole Scattering: A Unified Perspective
- Entanglement, anti-flatness, and nonlocal nonstabilizerness: a unified perspective from entanglement spectrum
- Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
BibTeX
@misc{castin2026unified,
title = {{A Unified Perspective on the Dynamics of Deep Transformers}},
author = {Valérie Castin and Pierre Ablin and José Antonio Carrillo and Gabriel Peyré},
howpublished = {Foundations of Computational Mathematics},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2501.18322},
eprint = {2501.18322},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05766600v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - A Unified Perspective on the Dynamics of Deep Transformers
AU - Valérie Castin
AU - Pierre Ablin
AU - José Antonio Carrillo
AU - Gabriel Peyré
PY - 2026
DA - 2026/09/28
PB - Foundations of Computational Mathematics
DO - 10.48550/arxiv.2501.18322
UR - https://hal.science/hal-05766600v1
AB - Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the iterative application of attention across layers induces complex dynamics that remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. Our first set of contributions focuses on compactly supported initial data. We show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system, thus generalizing and extending previous analysis to several variants of self-attention: multi-head attention, L2 attention, Sinkhorn attention, Sigmoid attention, and masked attention--leveraging a conditional Wasserstein framework. In a second set of contributions, we are the first to study non-compactly supported initial conditions, by focusing on Gaussian initial data. Again for different types of attention, we show that the Transformer PDE preserves the space of Gaussian measures, which allows us to analyze the Gaussian case theoretically and numerically to identify typical behaviors. This Gaussian analysis captures the evolution of data anisotropy through a deep Transformer. In particular, we highlight a clustering phenomenon that parallels previous results in the non-normalized discrete case.
SN - 1615-3375
SN - 1615-3383
ER -