Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
Abstract
We study row-softmax self-attention with independent Gaussian query and key weights in the proportional regime, at fixed inverse temperature. Hayase, Collins, and Karakida proved Gaussian equivalence for the empirical squared singular-value distribution after removal of the Perron direction. A global law alone does not exclude finitely many nonleading outliers. We prove that no such outliers persist: the rescaled squared singular value ℓ sk(A)2 converges in probability to the upper edge of their bulk law for every fixed k≥ 2. In fact, this convergence is uniform over any deterministic sublinear number of leading non-Perron indices. The proof uses an exact decomposition of the softmax normalization, conditions on the key matrix, identifies the conditional covariance exactly with a diagonally conjugated inner-product kernel, linearizes that kernel in operator norm, and applies the outside-support local law of Fan, Ma, Paquette, and Wang. A separate stability argument identifies the finite conditional deformed Marchenko-Pastur edge with the limiting bulk edge. We also derive a scalar formula for that edge throughout the proportional regime and recover the explicit square-model formula of Hayase, Collins, and Karakida, including its physical branch.
Related
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
- Notes on Uniform Policy Score Bounds and Lipschitz Continuity for Truncated Gaussian, Product Cauchy, Boltzmann, and Softmax Policies
- In-context classification: Calibration and noise in linear attention with softmax readout
- RoPE attention is an exact forward-pass gradient step with softmax intact
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
BibTeX
@misc{jerschow2026bulkedge,
title = {{Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention}},
author = {Alexander Jerschow},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.23949},
eprint = {2609.23949},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.23949},
}
RIS
TY - UNPB
TI - Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
AU - Alexander Jerschow
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.23949
UR - https://arxiv.org/abs/2609.23949
AB - We study row-softmax self-attention with independent Gaussian query and key weights in the proportional regime, at fixed inverse temperature. Hayase, Collins, and Karakida proved Gaussian equivalence for the empirical squared singular-value distribution after removal of the Perron direction. A global law alone does not exclude finitely many nonleading outliers. We prove that no such outliers persist: the rescaled squared singular value ℓ s_k(A)^2 converges in probability to the upper edge of their bulk law for every fixed k≥ 2. In fact, this convergence is uniform over any deterministic sublinear number of leading non-Perron indices. The proof uses an exact decomposition of the softmax normalization, conditions on the key matrix, identifies the conditional covariance exactly with a diagonally conjugated inner-product kernel, linearizes that kernel in operator norm, and applies the outside-support local law of Fan, Ma, Paquette, and Wang. A separate stability argument identifies the finite conditional deformed Marchenko-Pastur edge with the limiting bulk edge. We also derive a scalar formula for that edge throughout the proportional regime and recover the explicit square-model formula of Hayase, Collins, and Karakida, including its physical branch.
ER -