Skip to content
DraconiticResearch
Search Sources API
203,203 studies31 sourcesUpdated 17 min ago
PreprintCC BY 4.0arXiv

The Role of Feed-Forward Layers in Transformer Dynamics

  • Thomas Jacob Maranzatto
  • Semih Akkoc
  • Sennur Ulukus
Open at sourcePDFDOICite

Abstract

We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism, with the self-attention mechanism interpreted as an interacting particle system and the feed-forward layer as an independent control. Our main theoretical result establishes that the feed-forward network can steer the tokens arbitrarily close to consensus regardless of the key, query, and value matrices. Our result are easily extended to convergence to many clusters and to multi-head attention. We conduct numerical experiments to verify our results, and compare thresholding behavior from our theory to real-world LLMs.

Related

  1. From forward-backward sweep to backpropagation for optimal control in tumor-immune dynamics.
    Wenjing Zhang, Wandi Ding, Huaiping Zhu
    Journal of mathematical biologyPeer-reviewedCC BYFull textGeneral mathematicsEurope PMCPMC
  2. Second order estimates on transition layers for the Allen-Cahn equation with homogeneous Neumann boundary conditions: Part I - Boundary-orthogonal transition layers
    Akashdeep Dey, Wenkui Du, Davide Parise +1
    arXivPreprintCC BY 4.0Full textAnalysis & PDEsGeometry & topologyarXiv
  3. Content Exploration Beyond the Feed: Creator Supply and the Shared Corpus
    Yuanyuan Shen, Yiren Yan, Wenjie Li +1
    arXivPreprintCC BY-NC-ND 4.0Full textStatisticsarXiv
  4. Spectral stability for strong ideal-gas shock layers
    Jeffrey Humpherys
    arXivPreprintarXiv licenceFull textAnalysis & PDEsarXiv
  5. Ekman Boundary Layers Under Transport Noise
    Daniel Goodair, Szymon Sobczak
    arXivPreprintCC BY 4.0Full textAnalysis & PDEsProbabilityarXiv

Cite

BibTeXRIS
BibTeX
@misc{maranzatto2026role,
  title = {{The Role of Feed-Forward Layers in Transformer Dynamics}},
  author = {Thomas Jacob Maranzatto and Semih Akkoc and Sennur Ulukus},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.36230},
  eprint = {2609.36230},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.36230},
  note = {CC BY 4.0},
}
RIS
TY  - UNPB
TI  - The Role of Feed-Forward Layers in Transformer Dynamics
AU  - Thomas Jacob Maranzatto
AU  - Semih Akkoc
AU  - Sennur Ulukus
PY  - 2026
DA  - 2026/09/28
PB  - arXiv
DO  - 10.48550/arxiv.2609.36230
UR  - https://arxiv.org/abs/2609.36230
AB  - We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism, with the self-attention mechanism interpreted as an interacting particle system and the feed-forward layer as an independent control. Our main theoretical result establishes that the feed-forward network can steer the tokens arbitrarily close to consensus regardless of the key, query, and value matrices. Our result are easily extended to convergence to many clusters and to multi-head attention. We conduct numerical experiments to verify our results, and compare thresholding behavior from our theory to real-world LLMs.
ER  -