Attention by Synchronization in Coupled Oscillator Networks
Abstract
We address transformer attention on energy-constrained physical substrates. Softmax attention requires exponentiation and global reduction, operations with high energy cost on von Neumann hardware and no natural physical analog. We show that Kuramoto synchronization dynamics (which arise in electrical, mechanical, superconducting, and charge-density-wave oscillator arrays, among other physical systems) implement a well-defined attention operation. The resulting mechanism, \emph{fixed-query oscillator attention}, replaces softmax's arithmetic with the equilibration of a gradient flow on the sphere: queries are learned anchors fixed on the sphere, and free oscillators evolve under Kuramoto--Lohe dynamics until they settle at positions encoding attention weights via cosine similarity. Because the computation is equilibration, no global exponential normalization is needed. The fixed point is provably unique and globally attractive from almost every initial condition, a guarantee that holds across every physical realization. Empirically, at the minimal hardware configuration (oscillator dimension dosc = 2), oscillator attention matches softmax on keyword spotting, and on subject-verb agreement it trains more reliably while reaching softmax's accuracy. Softmax retains an advantage on causal language modeling, but the gap decays as a power law in dosc. The main objective of this work is not to replace softmax in software but to provide a mathematically grounded blueprint for accurate attention on physical substrates.
Related
- Remote synchronization and critical-like breakdown in star networks of oscillator populations
- Energy Transmission and Harmonic Propagation in Unidirectionally Coupled Plasma Oscillator Networks
- Computing with Coupled Oscillator Networks: Optimization, Memory, and Modern AI Primitives
- Label-Free Finite-Volume-Residual Training of Attention Graph Neural Networks for Coupled Thermo-Fluid Fields
- Kerr Induced Control of Synchronization and Quantum State Recovery in a Driven van der Pol Oscillator
BibTeX
@misc{pasqualetti2026attention,
title = {{Attention by Synchronization in Coupled Oscillator Networks}},
author = {Fabio Pasqualetti and Taosha Guo},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.12059},
eprint = {2606.12059},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.12059},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Attention by Synchronization in Coupled Oscillator Networks
AU - Fabio Pasqualetti
AU - Taosha Guo
PY - 2026
DA - 2026/09/09
PB - arXiv
DO - 10.48550/arxiv.2606.12059
UR - https://arxiv.org/abs/2606.12059
AB - We address transformer attention on energy-constrained physical substrates. Softmax attention requires exponentiation and global reduction, operations with high energy cost on von Neumann hardware and no natural physical analog. We show that Kuramoto synchronization dynamics (which arise in electrical, mechanical, superconducting, and charge-density-wave oscillator arrays, among other physical systems) implement a well-defined attention operation. The resulting mechanism, \emph{fixed-query oscillator attention}, replaces softmax's arithmetic with the equilibration of a gradient flow on the sphere: queries are learned anchors fixed on the sphere, and free oscillators evolve under Kuramoto--Lohe dynamics until they settle at positions encoding attention weights via cosine similarity. Because the computation is equilibration, no global exponential normalization is needed. The fixed point is provably unique and globally attractive from almost every initial condition, a guarantee that holds across every physical realization. Empirically, at the minimal hardware configuration (oscillator dimension d_osc = 2), oscillator attention matches softmax on keyword spotting, and on subject-verb agreement it trains more reliably while reaching softmax's accuracy. Softmax retains an advantage on causal language modeling, but the gap decays as a power law in d_osc. The main objective of this work is not to replace softmax in software but to provide a mathematically grounded blueprint for accurate attention on physical substrates.
ER -