Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Abstract
We study the nonequilibrium dynamics of a minimal recurrent transformer with N normalized tokens, Q=K=I, and a negative value map V=−I. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For d=2, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength γ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as N→∞ at finite fixed softmax sharpness β. Attention condensation instead emerges in the scaling regime β∼ N2. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For d=N→∞, simulations from Gaussian initial conditions provide evidence for a condensation transition at β=O(1), driven by dynamically generated finite overlap gaps. Depending on γ, the resulting phases include diffuse simplex-like states, consensus flips, condensed active routing with signatures of chaos, and fragmented cluster flips. These results establish temporal activity, attention condensation, and geometric clustering as distinct collective phenomena, and show that sparse attention can sustain persistent dynamics rather than freeze it.
Related
- Superconductivity in the Repulsive Hubbard Model on Different Geometries Induced by Density-Assisted Hopping
- Homogeneous attractive Bose-Einstein condensates with repulsive three-body interactions: the two-dimensional case
- Homogeneous attractive Bose-Einstein condensates with repulsive three-body interactions: the one-dimensional case
- Superconductivity via repulsive interaction in the model with electrons and holes
- Local Density Approximation and Other Limit Regimes for a Homogeneous Bose Gas with Repulsive Three-Body Interactions in Low-Dimensional Space
BibTeX
@misc{gao2026nonequilibrium,
title = {{Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality}},
author = {Qucheng Gao and Zuyi Yang and Xiao Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28448},
eprint = {2609.28448},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28448},
}
RIS
TY - UNPB
TI - Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
AU - Qucheng Gao
AU - Zuyi Yang
AU - Xiao Chen
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.28448
UR - https://arxiv.org/abs/2609.28448
AB - We study the nonequilibrium dynamics of a minimal recurrent transformer with N normalized tokens, Q=K=I, and a negative value map V=−I. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For d=2, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength γ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as N→∞ at finite fixed softmax sharpness β. Attention condensation instead emerges in the scaling regime β∼ N^2. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For d=N→∞, simulations from Gaussian initial conditions provide evidence for a condensation transition at β=O(1), driven by dynamically generated finite overlap gaps. Depending on γ, the resulting phases include diffuse simplex-like states, consensus flips, condensed active routing with signatures of chaos, and fragmented cluster flips. These results establish temporal activity, attention condensation, and geometric clustering as distinct collective phenomena, and show that sparse attention can sustain persistent dynamics rather than freeze it.
ER -