Generalization in Nonlinear Least Squares via Learned Feature Geometry
Abstract
We study the generalization of ridge-regularized nonlinear least-squares models via on-average algorithmic stability, deriving error bounds for local minimizers in terms of a data-dependent effective dimension that reflects the geometry of the gradient model at the trained parameters, through the empirical Jacobian Gram matrix and a residual-curvature term. In the linear case, where the curvature term vanishes, this recovers the classical effective dimension of the Jacobian kernel covariance, but evaluated at the trained model rather than at initialization as is typical in neural tangent kernel analyses. We further bound this effective dimension via covering complexity of the gradient features, leading to guarantees that depend on learned geometry rather than parameter count. In particular, for manifold-supported data and piecewise Lipschitz Jacobians, the bounds scale with intrinsic dimension, while for one-hidden-layer ReLU networks, the mechanism can be made explicit through counts of activation-stable regions. Experiments on synthetic manifolds, clustered distributions, and benchmark datasets illustrate trained-Jacobian compression, the tightness of the residual-curvature linearization, and agreement between the stability bound and observed generalization gaps. A key feature of our bounds is the simplicity of their derivation, which follows from first principles using the Brascamp-Lieb inequality under strongly log-concave noise.
Related
- Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory
- The algebraic geometry of 3-by-3 magic squares of squares
- Cross-validated feature selection using least-squares support vector machines for binary classification
- Contact Geometry and Covariance Deficits in Volume-Sampled Least Squares
- Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
BibTeX
@misc{kharel2026generalization,
title = {{Generalization in Nonlinear Least Squares via Learned Feature Geometry}},
author = {Ayub Kharel and Ilja Kuzborskij and Patrick Rebeschini and Yasin Abbasi-Yadkori},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.08799},
eprint = {2606.08799},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.08799},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Generalization in Nonlinear Least Squares via Learned Feature Geometry
AU - Ayub Kharel
AU - Ilja Kuzborskij
AU - Patrick Rebeschini
AU - Yasin Abbasi-Yadkori
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2606.08799
UR - https://arxiv.org/abs/2606.08799
AB - We study the generalization of ridge-regularized nonlinear least-squares models via on-average algorithmic stability, deriving error bounds for local minimizers in terms of a data-dependent effective dimension that reflects the geometry of the gradient model at the trained parameters, through the empirical Jacobian Gram matrix and a residual-curvature term. In the linear case, where the curvature term vanishes, this recovers the classical effective dimension of the Jacobian kernel covariance, but evaluated at the trained model rather than at initialization as is typical in neural tangent kernel analyses. We further bound this effective dimension via covering complexity of the gradient features, leading to guarantees that depend on learned geometry rather than parameter count. In particular, for manifold-supported data and piecewise Lipschitz Jacobians, the bounds scale with intrinsic dimension, while for one-hidden-layer ReLU networks, the mechanism can be made explicit through counts of activation-stable regions. Experiments on synthetic manifolds, clustered distributions, and benchmark datasets illustrate trained-Jacobian compression, the tightness of the residual-curvature linearization, and agreement between the stability bound and observed generalization gaps. A key feature of our bounds is the simplicity of their derivation, which follows from first principles using the Brascamp-Lieb inequality under strongly log-concave noise.
ER -