Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
Abstract
Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing analyses treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under L2 regularization. We identify a training-induced phase of the inference dynamics across layers: after clustering in the early layers, the token distribution can leave the clustered regime near the final layers. A single gradient-descent update already produces this escape, and minimizers of the training objective satisfy an exponential turnpike bound under which clustering persists through the interior layers. Our analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention, for which we establish a double-well landscape and a local Polyak-Łojasiewicz inequality. More broadly, our results point toward a training-aware mean-field theory in which training and inference dynamics of Transformers are treated together.
Related
- Closed escape path of smallest diameter in forest: dual formulation of Lebesgue's universal covering problem
- Transformers with CoT Discover Heavy-Ball Acceleration Through Training
- Information-Induced Training Geometry: Exact Reduction, Canonical Completion, and Structured Expressivity
- Asymptotic Behavior of Transformers
- Kinetic theory for Transformers and the lost-in-the-middle phenomenon
BibTeX
@misc{isobe2026traininginduced,
title = {{Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers}},
author = {Noboru Isobe and Daisuke Inoue and Masaaki Imaizumi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.07772},
eprint = {2605.07772},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.07772},
}
RIS
TY - UNPB
TI - Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
AU - Noboru Isobe
AU - Daisuke Inoue
AU - Masaaki Imaizumi
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2605.07772
UR - https://arxiv.org/abs/2605.07772
AB - Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing analyses treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under L^2 regularization. We identify a training-induced phase of the inference dynamics across layers: after clustering in the early layers, the token distribution can leave the clustered regime near the final layers. A single gradient-descent update already produces this escape, and minimizers of the training objective satisfy an exponential turnpike bound under which clustering persists through the interior layers. Our analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention, for which we establish a double-well landscape and a local Polyak-Łojasiewicz inequality. More broadly, our results point toward a training-aware mean-field theory in which training and inference dynamics of Transformers are treated together.
ER -