On the Statistical Mechanics of Attention
Abstract
We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
Related
- A liquid vapor phase transition in quantum statistical mechanics
- On the relaxation problem in statistical mechanics
- Statistical Mechanics of Spike Trains: Self-Consistent Activity Ensembles and Phase Transitions
- Entropic Fluctuations in Quantum Statistical Mechanics: A Review
- Dynamical Groups and Spherical Potentials in Classical Mechanics
BibTeX
@misc{hu2026statistical,
title = {{On the Statistical Mechanics of Attention}},
author = {Jerry Yao-Chieh Hu and Yu-An Chen and Maojiang Su and Han Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17347},
eprint = {2610.17347},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17347},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - On the Statistical Mechanics of Attention
AU - Jerry Yao-Chieh Hu
AU - Yu-An Chen
AU - Maojiang Su
AU - Han Liu
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.17347
UR - https://arxiv.org/abs/2610.17347
AB - We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
ER -