Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

Abstract

In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.

A discrete-continuous-time stochastic framework for attention-deficit/hyperactivity disorder-related symptom modeling and pharmacological intervention via methylphenidate
Oana-Maria-Catalina Crisan
arXiv18 Oct 2026PreprintarXiv licenceFull textProbabilityarXiv
Asymptotic Behavior of Transformers
Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3
arXiv14 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Inertial Primal-Dual Dynamics Methods Featuring Implicit Hessian-Driven Damping for Convex Optimization Problems in Continuous and Discrete Time
Xiangkai Sun, Zeying Gao, Liang He +1
arXiv29 Aug 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
On the well-posedness and long-time dynamics of a discrete coagulation--annihilation system
Ikjot Kaur, Saroj Si, Ankik Kumar Giri
arXiv16 Sep 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv
Equilibria and consensus in discrete-time opinion dynamics with mixed weighted-averaging and weighted-median mechanisms
Junkai Cheng, Wenjun Mei
arXiv12 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{altafini2026asymptotic,
  title = {{Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers}},
  author = {Claudio Altafini},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.13518},
  eprint = {2610.13518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.13518},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
AU  - Claudio Altafini
PY  - 2026
DA  - 2026/10/09
PB  - arXiv
DO  - 10.48550/arxiv.2610.13518
UR  - https://arxiv.org/abs/2610.13518
AB  - In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.
ER  -

Abstract

Related

Cite