Symmetry Breaking Is the Price of Finite Capacity
Abstract
If rotating a problem never changes its answer, one might expect its best model to be rotation-invariant; we show that this fails, predictably and by an exact amount, when capacity is limited. A symmetry groups a model's directions into blocks of equally strong directions, a symmetric model must use each block whole, and when the budget cuts a block every optimal model breaks the symmetry. For a Gaussian problem with a k-unit bilinear predictor, the log-loss price of symmetry is exactly (Uk−Ik)/(2σ2), the signal of the best k directions minus that of the best whole blocks. Because an attention head compares tokens through a similarity of rank at most its head dimension, whatever computation precedes it, the same price governs heads fitted to a symmetric similarity by squared error, and heads trained this way on three-dimensional directions break symmetry at exactly the predicted head dimensions, with the predicted rotation sensitivity (RMSE 0.004). Under softmax attention, flatter attention restores the predicted symmetric head where tested, while with peaked attention broken heads beat every whole-shell symmetric head at 16 of 20 head dimensions. We give a procedure that separates capacity-driven breaking from other causes.
Related
- Wavelet covariances under fixed finite observations: dilation symmetry breaking and distinct endpoint limits
- On the Role of Tie-Breaking Rules in the Convergence of Fictitious Play for Symmetric First-Price Auctions
- Dimension-dependent symmetry breaking in a diffusive nonlocal interaction model
- Full replica symmetry breaking in the Sherrington-Kirkpatrick model
- Uniqueness and Symmetry Breaking for Singular Q-curvature Equations in Rn
BibTeX
@misc{chlon2026symmetry,
title = {{Symmetry Breaking Is the Price of Finite Capacity}},
author = {Maggie Chlon and Leon Chlon},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36138},
eprint = {2610.36138},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36138},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Symmetry Breaking Is the Price of Finite Capacity
AU - Maggie Chlon
AU - Leon Chlon
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.36138
UR - https://arxiv.org/abs/2610.36138
AB - If rotating a problem never changes its answer, one might expect its best model to be rotation-invariant; we show that this fails, predictably and by an exact amount, when capacity is limited. A symmetry groups a model's directions into blocks of equally strong directions, a symmetric model must use each block whole, and when the budget cuts a block every optimal model breaks the symmetry. For a Gaussian problem with a k-unit bilinear predictor, the log-loss price of symmetry is exactly (U_k−I_k)/(2σ^2), the signal of the best k directions minus that of the best whole blocks. Because an attention head compares tokens through a similarity of rank at most its head dimension, whatever computation precedes it, the same price governs heads fitted to a symmetric similarity by squared error, and heads trained this way on three-dimensional directions break symmetry at exactly the predicted head dimensions, with the predicted rotation sensitivity (RMSE 0.004). Under softmax attention, flatter attention restores the predicted symmetric head where tested, while with peaked attention broken heads beat every whole-shell symmetric head at 16 of 20 head dimensions. We give a procedure that separates capacity-driven breaking from other causes.
ER -