Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs
Abstract
Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional representation. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) develop specialized neurons: individual neurons become strongly aligned with specific features relevant to a particular region of the input space. A substantial fraction of these neurons exhibit monosemanticity: they activate predominantly on inputs from the corresponding cluster. Thus, rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations. Furthermore, we show that this ability to specialize gives MLPs a provable data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.
Related
- Galois' Professor's Revenge
- Transfer Learning with Heterogeneous Feature Spaces in Linear Regression
- Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
- Geometric Feature Learning for Functional Data Valued on the Symmetric Positive Definite Manifold
- The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics
BibTeX
@misc{abedsoltan2026revenge,
title = {{Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs}},
author = {Amirhesam Abedsoltan and Enric Boix-Adsera and Fivos Kalogiannis and Mikhail Belkin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.24007},
eprint = {2608.24007},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.24007},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs
AU - Amirhesam Abedsoltan
AU - Enric Boix-Adsera
AU - Fivos Kalogiannis
AU - Mikhail Belkin
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2608.24007
UR - https://arxiv.org/abs/2608.24007
AB - Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional representation. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) develop specialized neurons: individual neurons become strongly aligned with specific features relevant to a particular region of the input space. A substantial fraction of these neurons exhibit monosemanticity: they activate predominantly on inputs from the corresponding cluster. Thus, rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations. Furthermore, we show that this ability to specialize gives MLPs a provable data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.
ER -