Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
Abstract
A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infinity, the NN training dynamics converge in probability to the solution of a deterministic limit equation involving a "Newton neural tangent kernel" (NNTK). Explicit rates characterizing this convergence are provided and, in the infinite-width limit, we prove that the NN converges exponentially fast to the target data (i.e., a global minimizer with zero loss). We show that this convergence is uniform across the frequency spectrum, addressing the spectral bias inherent in gradient descent. The eigenvalues of the NTK for gradient descent accumulate at zero, leading to slow convergence for target data with high-frequency components. In contrast, the NNTK has uniformly lower bounded eigenvalues if the regularization parameter is selected appropriately, allowing Newton's method to converge more quickly for data with high-frequency components. Mathematical challenges that need to be addressed include the implicit parameter update of the Newton method with a potentially indefinite Hessian matrix and the fact that the dimension of this linear system of equations tends to infinity as the NN width grows. This substantially complicates deriving the training dynamics in the overparameterized limit as well as proving the convergence of the finite-width dynamics thereto. Our analysis identifies a scaling formula for selecting the regularization parameter, which we show can vanish at a suitable rate as the NN width becomes larger. In addition, we prove that, for sufficiently large numbers of hidden units, the regularized Hessian remains positive definite during training and the Newton updates for individual NN parameters converge to zero, demonstrating that the model behaves as a linearization around the initialization.
Related
- Convergence of a Randomized Newton Method in Nonconvex Optimization
- Convergence Rates of Factorized Gradient Descent for Overparameterized Asymmetric Matrix Sensing with Balancing Regularization
- Convergence of stochastic gradient descent for learning linear neural networks
- Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence
- A samplewise backpropagation method for neural networks driven by fractional Brownian motion
BibTeX
@misc{riedl2026convergence,
title = {{Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit}},
author = {Konstantin Riedl and Konstantinos Spiliopoulos and Justin Sirignano},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.08352},
eprint = {2605.08352},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.08352},
}
RIS
TY - UNPB
TI - Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
AU - Konstantin Riedl
AU - Konstantinos Spiliopoulos
AU - Justin Sirignano
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2605.08352
UR - https://arxiv.org/abs/2605.08352
AB - A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infinity, the NN training dynamics converge in probability to the solution of a deterministic limit equation involving a "Newton neural tangent kernel" (NNTK). Explicit rates characterizing this convergence are provided and, in the infinite-width limit, we prove that the NN converges exponentially fast to the target data (i.e., a global minimizer with zero loss). We show that this convergence is uniform across the frequency spectrum, addressing the spectral bias inherent in gradient descent. The eigenvalues of the NTK for gradient descent accumulate at zero, leading to slow convergence for target data with high-frequency components. In contrast, the NNTK has uniformly lower bounded eigenvalues if the regularization parameter is selected appropriately, allowing Newton's method to converge more quickly for data with high-frequency components. Mathematical challenges that need to be addressed include the implicit parameter update of the Newton method with a potentially indefinite Hessian matrix and the fact that the dimension of this linear system of equations tends to infinity as the NN width grows. This substantially complicates deriving the training dynamics in the overparameterized limit as well as proving the convergence of the finite-width dynamics thereto. Our analysis identifies a scaling formula for selecting the regularization parameter, which we show can vanish at a suitable rate as the NN width becomes larger. In addition, we prove that, for sufficiently large numbers of hidden units, the regularized Hessian remains positive definite during training and the Newton updates for individual NN parameters converge to zero, demonstrating that the model behaves as a linearization around the initialization.
ER -