The Attention Within: Consensus Dynamics in Selective State Space Models
Abstract
Selective state space models (SSMs) have recently emerged as a compelling alternative to transformers, combining competitive performance with substantially improved inference efficiency. At each SSM layer, a sequence of hidden states are propagated by a recurrence, mixing information of different tokens. Despite using a different mechanism, this mixing plays a role analogous to attention in transformers. In fact, recent works have shown that the two architectures may be closer than they first appear, as this recurrence admits a formulation akin to linear attention. In transformers, attention is known to drive the tokens to cluster, i.e., to reach consensus, collapsing in the limit to a single direction. Thus, we ask: does the recurrence at the core of SSMs drive the tokens to consensus, as attention does in transformers? To answer this question, we take a dynamical systems perspective on SSMs, modeling the evolution of tokens across layers as an ordinary differential equation. By exploiting input-to-state stability arguments, we establish local exponential stability of the consensus equilibria and characterize their domain of attraction for time-varying weight matrices, a setting not addressed by previous results. We thereby show that the resemblance between SSMs and transformers does run deeper: the recurrence at the core of SSMs aggregates tokens just as attention does. Numerical experiments on a pretrained Mamba-2 model point to the output gate as the component that regulates the extent of this consensus, preventing the tokens from reaching it in full.
Related
- Input-to-State Stability Framework for Fully Distributed Primal-Dual Dynamics for Quadratic GNEPs Without Multiplier Consensus
- Input-to-Output Stability and Consensus Control of Opinion Dynamics on Graph State-Space
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
- Equilibria and consensus in discrete-time opinion dynamics with mixed weighted-averaging and weighted-median mechanisms
- Nearly-Optimal Time-Sample Trade-off in Consensus Dynamics
BibTeX
@misc{silvestre2026attention,
title = {{The Attention Within: Consensus Dynamics in Selective State Space Models}},
author = {João Pedro Silvestre and Álvaro Rodríguez Abella and Paulo Tabuada},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.17997},
eprint = {2609.17997},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.17997},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - The Attention Within: Consensus Dynamics in Selective State Space Models
AU - João Pedro Silvestre
AU - Álvaro Rodríguez Abella
AU - Paulo Tabuada
PY - 2026
DA - 2026/09/16
PB - arXiv
DO - 10.48550/arxiv.2609.17997
UR - https://arxiv.org/abs/2609.17997
AB - Selective state space models (SSMs) have recently emerged as a compelling alternative to transformers, combining competitive performance with substantially improved inference efficiency. At each SSM layer, a sequence of hidden states are propagated by a recurrence, mixing information of different tokens. Despite using a different mechanism, this mixing plays a role analogous to attention in transformers. In fact, recent works have shown that the two architectures may be closer than they first appear, as this recurrence admits a formulation akin to linear attention. In transformers, attention is known to drive the tokens to cluster, i.e., to reach consensus, collapsing in the limit to a single direction. Thus, we ask: does the recurrence at the core of SSMs drive the tokens to consensus, as attention does in transformers? To answer this question, we take a dynamical systems perspective on SSMs, modeling the evolution of tokens across layers as an ordinary differential equation. By exploiting input-to-state stability arguments, we establish local exponential stability of the consensus equilibria and characterize their domain of attraction for time-varying weight matrices, a setting not addressed by previous results. We thereby show that the resemblance between SSMs and transformers does run deeper: the recurrence at the core of SSMs aggregates tokens just as attention does. Numerical experiments on a pretrained Mamba-2 model point to the output gate as the component that regulates the extent of this consensus, preventing the tokens from reaching it in full.
ER -