Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention
Abstract
We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group G, then G can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For G=D4 acting on C copies of the regular representation as the token feature space, we show that there are Ω(C64) components for eight attention heads. We illustrate this with experiments on artificial tasks and show that the predicted head-cluster structures can arise naturally when the attention layer is pressured to become equivariant.
Related
- On the Trade-off Between Information Loss and Generalization in Sparse Attention
- Differentiable FEM versus PINN for pavement layer inverse analysis: enforcing physics as hard or soft constraints
- High-Order-Accurate Continuity Enforcing Nyström Discretization of 3D Maxwell Combined Field Integral Equations
- Non-induced inherent graphs
- Invariant means, coset correct means and the Hahn-Banach Theorem
BibTeX
@misc{ong2026inherent,
title = {{Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention}},
author = {Tīkun Ông and Georg Bökman},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.30417},
eprint = {2608.30417},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.30417},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention
AU - Tīkun Ông
AU - Georg Bökman
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2608.30417
UR - https://arxiv.org/abs/2608.30417
AB - We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group G, then G can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For G=D_4 acting on C copies of the regular representation as the token feature space, we show that there are Ω(C^64) components for eight attention heads. We illustrate this with experiments on artificial tasks and show that the predicted head-cluster structures can arise naturally when the attention layer is pressured to become equivariant.
ER -