Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
Abstract
Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for Wd(q)=∑j=1dσ(qj): its d two-atom heads use 2d head-atom terms, whereas its minimum single-head representation has exactly 2d atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for every fixed uniform error below 1/2 on ℝd, and the threshold is sharp. The proof uses exposed-key normal cones; exact equality on any nonempty open query domain inherits the bound. The resulting atom bound for a fixed head count is tight when that count divides d. A normalized family retains its exponential exact minimum while approaching a constant on fixed-radius query balls. For pretrained-model caches, a classical affine-rank bound and a certified instance exclude exact vector preservation at a specified atom count. Learned-cache comparisons on two disjoint banks of 64 contexts show budget-dependent reversals and calibration effects.
Related
- RoPE attention is an exact forward-pass gradient step with softmax intact
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
- In-context classification: Calibration and noise in linear attention with softmax readout
- Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
- Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
BibTeX
@misc{usami2026exact,
title = {{Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms}},
author = {Hiroyasu Usami and Keisuke Hara and Ayato Tsuboi and Naohiko Matsuda},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.02943},
eprint = {2611.02943},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.02943},
}
RIS
TY - UNPB
TI - Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
AU - Hiroyasu Usami
AU - Keisuke Hara
AU - Ayato Tsuboi
AU - Naohiko Matsuda
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.02943
UR - https://arxiv.org/abs/2611.02943
AB - Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for W_d(q)=∑_j=1^dσ(q_j): its d two-atom heads use 2d head-atom terms, whereas its minimum single-head representation has exactly 2^d atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for every fixed uniform error below 1/2 on ℝ^d, and the threshold is sharp. The proof uses exposed-key normal cones; exact equality on any nonempty open query domain inherits the bound. The resulting atom bound for a fixed head count is tight when that count divides d. A normalized family retains its exponential exact minimum while approaching a constant on fixed-radius query balls. For pretrained-model caches, a classical affine-rank bound and a certified instance exclude exact vector preservation at a specified atom count. Learned-cache comparisons on two disjoint banks of 64 contexts show budget-dependent reversals and calibration effects.
ER -