Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
Abstract
An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a \emph{single scalar output}, providing a normal form for large-step GD in a neighbourhood of an \emph{isolated} flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with \emph{vector-valued outputs} (including regression with arbitrarily many observations), and (2) to a neighbourhood of a \emph{manifold} of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.
Related
- PL conditions do not guarantee convergence of gradient descent-ascent dynamics
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- Gradient descent reaches arbitrary accuracy in a fixed number of large steps on multiclass logistic regression
- Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization
- Local Algebraic Stability of Wasserstein Gradient Descent-Ascent and Mean Field Langevin Dynamics without Spectral Gap
BibTeX
@misc{macdonald2026dynamics,
title = {{Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima}},
author = {Lachlan Ewen MacDonald and René Vidal},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.08380},
eprint = {2607.08380},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.08380},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
AU - Lachlan Ewen MacDonald
AU - René Vidal
PY - 2026
DA - 2026/09/06
PB - arXiv
DO - 10.48550/arxiv.2607.08380
UR - https://arxiv.org/abs/2607.08380
AB - An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a \emph{single scalar output}, providing a normal form for large-step GD in a neighbourhood of an \emph{isolated} flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with \emph{vector-valued outputs} (including regression with arbitrarily many observations), and (2) to a neighbourhood of a \emph{manifold} of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.
ER -