Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking
Abstract
Learning in a neural network takes place on a high dimensional, non-convex loss landscape, the Hessian curvature of which dictates the learning dynamics. Despite its high dimensionality, the large number of architectural symmetries explains some of the geometric of this landscape, in particular flat directions, i.e. near-zero modes of the Hessian. These architectural symmetries have therefore been suggested as an organizing principle of deep learning. It remains an open question, though, wether there are additional organizing principles in the remaining degrees of freedom in the learning dynamic. Here we identify task-dependent contingent symmetries as such an additional organizing principle. We show that non-flat learning directions emerge due to explicit breaking of these contingent symmetries. We isolate this symmetry in an analytically solvable minimal model. Furthermore, we show that, by tuning an L2 penalty, signatures of the symmetry breaking become measurable. As a consequence of the same mechanism, adding noise to the data drives the model through a sequence of broken symmetry recoveries in the form of Baik-Ben-Arous-Péché (BBP) transitions. We demonstrate analytically for linear and numerically for deep linear networks that the contingent symmetry group remains explicit, and that the symmetry-breaking mechanism identifies which Hessian modes carry task-specific information. These signatures are measurable without a symmetry-group description and, hence, offer a route to study symmetry breaking as a more general principle in nonlinear networks and other learning systems.
Related
- Can Spontaneous Time-Translation Symmetry Breaking Be Quantized Without Explicit Branches?
- Dimension-dependent symmetry breaking in a diffusive nonlocal interaction model
- Full replica symmetry breaking in the Sherrington-Kirkpatrick model
- Uniqueness and Symmetry Breaking for Singular Q-curvature Equations in Rn
- Symmetry breaking differential operators and Discrete Series
BibTeX
@misc{ersoy2026informative,
title = {{Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking}},
author = {Ibrahim Talha Ersoy and Björn Ladewig and Karoline Wiesner},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.00745},
eprint = {2611.00745},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.00745},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking
AU - Ibrahim Talha Ersoy
AU - Björn Ladewig
AU - Karoline Wiesner
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.00745
UR - https://arxiv.org/abs/2611.00745
AB - Learning in a neural network takes place on a high dimensional, non-convex loss landscape, the Hessian curvature of which dictates the learning dynamics. Despite its high dimensionality, the large number of architectural symmetries explains some of the geometric of this landscape, in particular flat directions, i.e. near-zero modes of the Hessian. These architectural symmetries have therefore been suggested as an organizing principle of deep learning. It remains an open question, though, wether there are additional organizing principles in the remaining degrees of freedom in the learning dynamic. Here we identify task-dependent contingent symmetries as such an additional organizing principle. We show that non-flat learning directions emerge due to explicit breaking of these contingent symmetries. We isolate this symmetry in an analytically solvable minimal model. Furthermore, we show that, by tuning an L2 penalty, signatures of the symmetry breaking become measurable. As a consequence of the same mechanism, adding noise to the data drives the model through a sequence of broken symmetry recoveries in the form of Baik-Ben-Arous-Péché (BBP) transitions. We demonstrate analytically for linear and numerically for deep linear networks that the contingent symmetry group remains explicit, and that the symmetry-breaking mechanism identifies which Hessian modes carry task-specific information. These signatures are measurable without a symmetry-group description and, hence, offer a route to study symmetry breaking as a more general principle in nonlinear networks and other learning systems.
ER -