Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
Abstract
We study the generalization of spectral gradient descent (SpecGD) in overparameterized matrix classification with corrupted labels. Each input combines a shared low-rank signal with a rank-one sample-specific perturbation, referred to as a shortcut, that enables memorization but does not generalize. We contrast collapsed shortcuts, which share a singular direction, with dispersed shortcuts, which occupy distinct singular directions. Changing only this geometry can reverse the relative generalization of GD and SpecGD: collapsed shortcuts can favor SpecGD, while dispersed shortcuts can favor GD. In the dispersed regime, exact shortcut orthogonality eliminates the signal from the late-stage SpecGD direction, while vanishing random correlations collectively generate a small but generalization-relevant signal through a second-order effect. To identify the direction selected by SpecGD, which the spectral max-margin problem alone does not determine, we combine a refined analysis of its dual with the exponentiated-gradient dynamics of normalized loss weights. Finally, we show that a single SpecGD step can already interpolate and generalize well, while continued training converges to a direction with substantially worse generalization.
Related
- Average Energy Dissipation Rates of Implicit-Explicit Runge-Kutta Methods for Gradient Flow Problems
- Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks
- Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms
- A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
- From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
BibTeX
@misc{braun2026overfitting,
title = {{Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias}},
author = {Guillaume Braun and Ichiro Hashimoto and Masaaki Imaizumi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32270},
eprint = {2609.32270},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32270},
}
RIS
TY - UNPB
TI - Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
AU - Guillaume Braun
AU - Ichiro Hashimoto
AU - Masaaki Imaizumi
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2609.32270
UR - https://arxiv.org/abs/2609.32270
AB - We study the generalization of spectral gradient descent (SpecGD) in overparameterized matrix classification with corrupted labels. Each input combines a shared low-rank signal with a rank-one sample-specific perturbation, referred to as a shortcut, that enables memorization but does not generalize. We contrast collapsed shortcuts, which share a singular direction, with dispersed shortcuts, which occupy distinct singular directions. Changing only this geometry can reverse the relative generalization of GD and SpecGD: collapsed shortcuts can favor SpecGD, while dispersed shortcuts can favor GD. In the dispersed regime, exact shortcut orthogonality eliminates the signal from the late-stage SpecGD direction, while vanishing random correlations collectively generate a small but generalization-relevant signal through a second-order effect. To identify the direction selected by SpecGD, which the spectral max-margin problem alone does not determine, we combine a refined analysis of its dual with the exponentiated-gradient dynamics of normalized loss weights. Finally, we show that a single SpecGD step can already interpolate and generalize well, while continued training converges to a direction with substantially worse generalization.
ER -