Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking
Abstract
Hessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated outliers. This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference. Modifications, including changes to the architecture, data distribution, or parameter metric, expose a nearby reference configuration whose Hessian exhibits rich invariances-ones not accounted for by weight symmetries. There, symmetry enables a precise description of the spectra, forcing high-dimensional kernels and eigenvalues of large multiplicity. Returning to the original configuration breaks the Hessian symmetry and thereby produces the observed hierarchy of clusters and outliers. The framework is developed in some generality, with a detailed analysis of three-layer ReLU networks and applications to convolutional, graph, and transformer models, as well as to the NTK. The same mechanism is further shown to yield analogous spectral structures in layerwise Hessians and the Gauss-Newton matrix.
Related
- The Projected Hessian Quantification Theorem: Exact Duality For Constrained Eigenvalues
- The Hessian of Planar Central Configurations in Pair Space: Decomposition, Morse Index and Symmetry Reduction
- Dimension-dependent symmetry breaking in a diffusive nonlocal interaction model
- Full replica symmetry breaking in the Sherrington-Kirkpatrick model
- Uniqueness and Symmetry Breaking for Singular Q-curvature Equations in Rn
BibTeX
@misc{arjevani2026eigenvalues,
title = {{Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking}},
author = {Yossi Arjevani},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.09919},
eprint = {2610.09919},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.09919},
}
RIS
TY - UNPB
TI - Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking
AU - Yossi Arjevani
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.09919
UR - https://arxiv.org/abs/2610.09919
AB - Hessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated outliers. This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference. Modifications, including changes to the architecture, data distribution, or parameter metric, expose a nearby reference configuration whose Hessian exhibits rich invariances-ones not accounted for by weight symmetries. There, symmetry enables a precise description of the spectra, forcing high-dimensional kernels and eigenvalues of large multiplicity. Returning to the original configuration breaks the Hessian symmetry and thereby produces the observed hierarchy of clusters and outliers. The framework is developed in some generality, with a detailed analysis of three-layer ReLU networks and applications to convolutional, graph, and transformer models, as well as to the NTK. The same mechanism is further shown to yield analogous spectral structures in layerwise Hessians and the Gauss-Newton matrix.
ER -