A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Abstract
Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training. More precisely, we study a broad class of neural networks trained under the ℓ2 loss by gradient descent (GD) with weight decay, and prove the convergence of GD to a neighbourhood of the global minimizers of the empirical loss. By partitioning the space based on the input data, we then decompose the population error into data error, optimization error, and prediction variation error, and bound them separately. In particular, for the prediction variation error, which measures the oscillations of the learned function, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds for its evolution along the training trajectory. These bounds yield two important implications: a necessary condition of improved generalization explains differences in layerwise generalization behavior; a sufficient condition describes delayed generalization and provides a theoretical characterization of grokking.
Related
- Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms
- Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
- Convergence of stochastic gradient descent for learning linear neural networks
- Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks
- PL conditions do not guarantee convergence of gradient descent-ascent dynamics
BibTeX
@misc{wang2026theoretical,
title = {{A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay}},
author = {Yuqing Wang and Ioannis G. Kevrekidis and Mikhail Belkin},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.07755},
eprint = {2609.07755},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.07755},
}
RIS
TY - UNPB
TI - A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
AU - Yuqing Wang
AU - Ioannis G. Kevrekidis
AU - Mikhail Belkin
PY - 2026
DA - 2026/09/07
PB - arXiv
DO - 10.48550/arxiv.2609.07755
UR - https://arxiv.org/abs/2609.07755
AB - Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training. More precisely, we study a broad class of neural networks trained under the ℓ^2 loss by gradient descent (GD) with weight decay, and prove the convergence of GD to a neighbourhood of the global minimizers of the empirical loss. By partitioning the space based on the input data, we then decompose the population error into data error, optimization error, and prediction variation error, and bound them separately. In particular, for the prediction variation error, which measures the oscillations of the learned function, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds for its evolution along the training trajectory. These bounds yield two important implications: a necessary condition of improved generalization explains differences in layerwise generalization behavior; a sufficient condition describes delayed generalization and provides a theoretical characterization of grokking.
ER -