PreprintarXiv licencearXiv
Gaussian Equivalence for Multi-Head Self-Attention
Abstract
A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.
Related
- Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention
- Multi-Head Self Attention is a Parameter Identification Mechanism
- RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention
- Gaussian equivalence for CNN with non-Gaussian weights
- Correlated Gaussian Moments Determine Polynomials up to Orthogonal Equivalence
BibTeX
@misc{hayase2026gaussian,
title = {{Gaussian Equivalence for Multi-Head Self-Attention}},
author = {Tomohiro Hayase and Ryo Karakida},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10033},
eprint = {2610.10033},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10033},
}
RIS
TY - UNPB
TI - Gaussian Equivalence for Multi-Head Self-Attention
AU - Tomohiro Hayase
AU - Ryo Karakida
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10033
UR - https://arxiv.org/abs/2610.10033
AB - A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.
ER -