Do Neural Models Prefer Their Submodels?
Abstract
A neural model is the set of functions that can be expressed by a given neural network architecture. Neural models contain natural submodels, such as those obtained by setting some neurons to be inactive. We study the geometry of neural models and ask whether they prefer their submodels. By this we mean whether local minimizers of the loss function lie in a given submodel. We introduce a geometric framework to determine if this occurs. For shallow multilayer perceptrons, we show that subnetworks with inactive hidden neurons are never preferred. For deep multilayer perceptrons, we give conditions under which subnetworks with inactive hidden neurons are not preferred. For lightning self-attention, lower-rank submodels are usually not preferred, but we identify examples where they are because they lie on a boundary of the model. Our results show that, aside from a few exceptions, preference for submodels cannot be explained by the geometry of the neural model, and must instead come from other sources such as training dynamics.
Related
- Active learning confidence measures for coupling strategies in digital twins integrating simulation and data-driven submodels
- More on expressibility of satisfiability in submodels and extensions
- PREFER: Personalized Review Summarization with Online Preference Learning
- Neural Ideals and Neural Codes: An Algebraic Framework for Neural Network Classification and Feature Interpretation
- Singular perturbations and hierarchical learning in two-layer neural networks
BibTeX
@misc{ribot2026neural,
title = {{Do Neural Models Prefer Their Submodels?}},
author = {Alvaro Ribot and Yulia Alexandr and Anna Seigal},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28439},
eprint = {2610.28439},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28439},
}
RIS
TY - UNPB
TI - Do Neural Models Prefer Their Submodels?
AU - Alvaro Ribot
AU - Yulia Alexandr
AU - Anna Seigal
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28439
UR - https://arxiv.org/abs/2610.28439
AB - A neural model is the set of functions that can be expressed by a given neural network architecture. Neural models contain natural submodels, such as those obtained by setting some neurons to be inactive. We study the geometry of neural models and ask whether they prefer their submodels. By this we mean whether local minimizers of the loss function lie in a given submodel. We introduce a geometric framework to determine if this occurs. For shallow multilayer perceptrons, we show that subnetworks with inactive hidden neurons are never preferred. For deep multilayer perceptrons, we give conditions under which subnetworks with inactive hidden neurons are not preferred. For lightning self-attention, lower-rank submodels are usually not preferred, but we identify examples where they are because they lie on a boundary of the model. Our results show that, aside from a few exceptions, preference for submodels cannot be explained by the geometry of the neural model, and must instead come from other sources such as training dynamics.
ER -