Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
Abstract
Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity, and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.
Related
- Dual-domain self-supervised feature alignment via spectral–spatial representation learning for deepfake anomaly detection
- Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning
- Singular perturbations and hierarchical learning in two-layer neural networks
- Learning Long-Term Stable Operator Inference Reduced-Order Models of Fluid Flows through Online Spatial Filtering
- Transfer Learning with Heterogeneous Feature Spaces in Linear Regression
BibTeX
@misc{dandi2026deep,
title = {{Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning}},
author = {Yatin Dandi and Matteo Vilucchio and Luca Arnaboldi and Hugo Tabanelli and Florent Krzakala},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.13612},
eprint = {2605.13612},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.13612},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
AU - Yatin Dandi
AU - Matteo Vilucchio
AU - Luca Arnaboldi
AU - Hugo Tabanelli
AU - Florent Krzakala
PY - 2026
DA - 2026/09/04
PB - arXiv
DO - 10.48550/arxiv.2605.13612
UR - https://arxiv.org/abs/2605.13612
AB - Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity, and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.
ER -