A Smoothed Discrepancy Principle for Random Feature Methods and Neural Networks
Abstract
We study data-driven early stopping for spectral regularisation methods in the classical non-parametric regression setting. Building on the discrepancy principle, we propose a multi-scale stopping rule that applies to general kernel estimators and show that, unlike previous approaches, it achieves full adaptivity over all smoothness levels in the well-specified case. A key contribution of our work is an extension based on random feature approximations, which reduces computational cost on large datasets while preserving minimax-optimal statistical guarantees. Our procedure not only selects an optimal stopping time but also provides a fully data-driven choice of the number of random features needed to achieve optimal rates. Through the established connection between random features and neural networks in the neural tangent kernel regime, our method further yields a principled, data-driven recommendation for the network width. We prove that the resulting simultaneously chosen width and stopping time allow neural networks to attain minimax-optimal learning rates without prior knowledge of smoothness or capacity parameters.
Related
- Early stopping of stochastic variance reduced gradient for linear inverse problems by the discrepancy principle
- Minimax-Optimal Early Stopping for Continuous-Time SGD via the Discrepancy Principle
- Composite and Stochastic Smoothed Decentralized Bregman Proximal-Gradient Methods
- Learning Coarse Spaces for Two-Grid Methods from Smoothed Errors with Applications to High-Contrast Porous Media Flow
- Comparison Principle and Semi-Lagrangian Methods for a Hamilton--Jacobi--Bellman System with Recursive Utility
BibTeX
@misc{nguyen2026smoothed,
title = {{A Smoothed Discrepancy Principle for Random Feature Methods and Neural Networks}},
author = {Mike Nguyen and Nicole Mücke},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.21017},
eprint = {2609.21017},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.21017},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Smoothed Discrepancy Principle for Random Feature Methods and Neural Networks
AU - Mike Nguyen
AU - Nicole Mücke
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2609.21017
UR - https://arxiv.org/abs/2609.21017
AB - We study data-driven early stopping for spectral regularisation methods in the classical non-parametric regression setting. Building on the discrepancy principle, we propose a multi-scale stopping rule that applies to general kernel estimators and show that, unlike previous approaches, it achieves full adaptivity over all smoothness levels in the well-specified case. A key contribution of our work is an extension based on random feature approximations, which reduces computational cost on large datasets while preserving minimax-optimal statistical guarantees. Our procedure not only selects an optimal stopping time but also provides a fully data-driven choice of the number of random features needed to achieve optimal rates. Through the established connection between random features and neural networks in the neural tangent kernel regime, our method further yields a principled, data-driven recommendation for the network width. We prove that the resulting simultaneously chosen width and stopping time allow neural networks to attain minimax-optimal learning rates without prior knowledge of smoothness or capacity parameters.
ER -