Removing spurious minima for planar features by skip connections
Abstract
Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.
Related
- No Spurious Local Minima in Full-Order Linear Quadratic Gaussian Control
- W01,1 minima of noncoercive functionals
- Une méthode d'optimisation stochastique pour évaluer des minima à ε près
- Local minima of the empirical risk in high dimension: General theorems and convex examples
- The Singular Set of Minima of Multiple Integrals
BibTeX
@misc{zimmermann2026removing,
title = {{Removing spurious minima for planar features by skip connections}},
author = {Jakob Paul Zimmermann and Andrei Balakin and Moritz Grillo and Georg Loho},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01728},
eprint = {2610.01728},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01728},
}
RIS
TY - UNPB
TI - Removing spurious minima for planar features by skip connections
AU - Jakob Paul Zimmermann
AU - Andrei Balakin
AU - Moritz Grillo
AU - Georg Loho
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.01728
UR - https://arxiv.org/abs/2610.01728
AB - Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.
ER -