Towards a mathematical theory of superposition
Abstract
We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector x of active features is encoded through an overcomplete dictionary W, and feature recovery is performed by applying ReLU(W⊤ W x+b) with an appropriate bias vector b. We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order d/log n. In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with n>d+1, we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.
Related
- Hanslick's Musical Formalism as a Mathematical Structure - Examining meaning in Music through the Lens of Category Theory
- A Mathematical Justification of the Theory of Linearly Elastic Plates
- Some New Notions of Mathematical Integral Inequalities: Theory and Applications
- A Mathematical Theory of Near-Field Super-Resolution
- Algebraic Theory of Inductive Mathematical Modeling: From Experimental Observations to State-Space Representations
BibTeX
@misc{ivanitskiy2026towards,
title = {{Towards a mathematical theory of superposition}},
author = {Michael I. Ivanitskiy and John Jasper and Emily J. King and Dustin G. Mixon},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.27540},
eprint = {2608.27540},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.27540},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Towards a mathematical theory of superposition
AU - Michael I. Ivanitskiy
AU - John Jasper
AU - Emily J. King
AU - Dustin G. Mixon
PY - 2026
DA - 2026/08/27
PB - arXiv
DO - 10.48550/arxiv.2608.27540
UR - https://arxiv.org/abs/2608.27540
AB - We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector x of active features is encoded through an overcomplete dictionary W, and feature recovery is performed by applying ReLU(W^⊤ W x+b) with an appropriate bias vector b. We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order d/log n. In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with n>d+1, we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.
ER -