Exceptional Singularities in Polynomial Neural Networks
Abstract
The singularities of a neurovariety have so far always turned out to lie at subnetworks: functions already computed by a smaller architecture of the same kind. This is what makes them readable as a bias of training towards simpler models, and whether they exhaust the singular locus is left open in the literature. We show that they do not. There are singular points at which every neuron is active, no two neurons are redundant, and the parametrisation is unramified, so that nothing visible at a single parameter signals the singularity. We call these exceptional, and we show that, after quotienting by the group of architectural symmetries, which acts freely on the full parameters, they can arise in only three ways: a rank drop of the differential of the parametrisation, a disconnected fibre, or a failure of properness. A small dimensional example is a fully connected network with two hidden layers of width two and quadratic activations. All the examples we produce are of the second kind, so that what fails at an exceptional singularity is not the simplicity of the model but the possibility of recovering the parameter, up to symmetry, from the function it computes.
Related
- Minimal Filling Architectures for Polynomial Neural Networks
- Algebraic Expressivity Certificates for Shallow Polynomial Neural Networks
- Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework
- INI-VPINN: A Variational Physics-Informed Neural Network with Implicit Neumann and Interface Handling for Multi-Material Domains with Geometric Singularities
- Polynomial Residual Networks are Generically Identifiable
BibTeX
@misc{graziani2026exceptional,
title = {{Exceptional Singularities in Polynomial Neural Networks}},
author = {Giacomo Graziani},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.18440},
eprint = {2606.18440},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.18440},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Exceptional Singularities in Polynomial Neural Networks
AU - Giacomo Graziani
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2606.18440
UR - https://arxiv.org/abs/2606.18440
AB - The singularities of a neurovariety have so far always turned out to lie at subnetworks: functions already computed by a smaller architecture of the same kind. This is what makes them readable as a bias of training towards simpler models, and whether they exhaust the singular locus is left open in the literature. We show that they do not. There are singular points at which every neuron is active, no two neurons are redundant, and the parametrisation is unramified, so that nothing visible at a single parameter signals the singularity. We call these exceptional, and we show that, after quotienting by the group of architectural symmetries, which acts freely on the full parameters, they can arise in only three ways: a rank drop of the differential of the parametrisation, a disconnected fibre, or a failure of properness. A small dimensional example is a fully connected network with two hidden layers of width two and quadratic activations. All the examples we produce are of the second kind, so that what fails at an exceptional singularity is not the simplicity of the model but the possibility of recovering the parameter, up to symmetry, from the function it computes.
ER -