High-probability guarantees for linear accessibility in feature superposition
Abstract
Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly (d=Oε(k log m)) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.
Related
- High-Probability Guarantees for SGD under β-Heavy-Tailed Gradient Noise
- MCIR: A Feature Dependence-Aware Explainability Method with Reliability Guarantees
- posterior-landscape: morphological feature finding in two-dimensional probability densities
- Minimum-Energy Control for Stochastic Linear Systems: A Data-Driven Approach with Finite-Sample Guarantees
- Boundary geometry and linear accessibility of functions with positive real derivative
BibTeX
@misc{vompa2026highprobability,
title = {{High-probability guarantees for linear accessibility in feature superposition}},
author = {Enrico Vompa},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.09556},
eprint = {2609.09556},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09556},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - High-probability guarantees for linear accessibility in feature superposition
AU - Enrico Vompa
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.09556
UR - https://arxiv.org/abs/2609.09556
AB - Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly (d=O_ε(k log m)) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.
ER -