Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention
Abstract
We study nonparametric regression using an over-parameterized two-layer ReLU neural network with fixed covariance-based channel attention, trained by gradient descent with both layers updated simultaneously. The spherical covariates follow an arbitrary fixed non-atomic distribution, and the target function lies in an interpolation space with spectral bias widely studied in the statistical learning literature. We prove that early-stopped training achieves regression risk cO(epsn2), where epsn is the critical population rate of the attention kernel Kattn. Its positive eigenvalues are λjs', where λj are those of the ReLU feature covariance kernel K and s'=2s+3 for s≥1. Under polynomial eigenvalue decay λjasymp j−2α with α>1/2, the regression risk bound cO(epsn2) specializes to the minimax optimal rate cO(n−2αs'/(2αs'+1)). Our analysis is based on two new technical results on the optimization and generalization of the attention network. First, the neural network function after every gradient descent iteration admits a decomposition into a function with bounded Reproducing Kernel Hilbert Space (RKHS) norm and a uniformly small residual. Second, local Rademacher complexity converts this decomposition into a regression risk bound. Kernel gradient descent on the same ball attains the same rate, while our theorem gives explicit sufficient width and auxiliary-sample conditions for simultaneous network training. Our analysis is further extended to covariates supported on RRd through a novel normalization method. Under a sub-Gaussian covariate distribution, the attention network achieves a nearly optimal regression risk cO((log n)n−s'(d+3)/(s'(d+3)+d)) without a density assumption.
Related
- A Gaussian Process Regression-Enhanced Two-Sweep Parameterized Power Method for Computing Extreme Eigenpairs
- The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression
- Recursive moment updates for scalable Bayesian nonparametric regression
- Choosing the penalty in nonparametric regression: short and long-range dependence
- Implementation of Linear Regression and Linear Interpolation using Reaction Networks
BibTeX
@misc{yang2026sharp,
title = {{Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention}},
author = {Yingzhen Yang and Ping Li},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01642},
eprint = {2611.01642},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01642},
}
RIS
TY - UNPB
TI - Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention
AU - Yingzhen Yang
AU - Ping Li
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01642
UR - https://arxiv.org/abs/2611.01642
AB - We study nonparametric regression using an over-parameterized two-layer ReLU neural network with fixed covariance-based channel attention, trained by gradient descent with both layers updated simultaneously. The spherical covariates follow an arbitrary fixed non-atomic distribution, and the target function lies in an interpolation space with spectral bias widely studied in the statistical learning literature. We prove that early-stopped training achieves regression risk cO(eps_n^2), where eps_n is the critical population rate of the attention kernel Kattn. Its positive eigenvalues are λ_j^s', where λ_j are those of the ReLU feature covariance kernel K and s'=2s+3 for s≥1. Under polynomial eigenvalue decay λ_jasymp j^−2α with α>1/2, the regression risk bound cO(eps_n^2) specializes to the minimax optimal rate cO(n^−2αs'/(2αs'+1)). Our analysis is based on two new technical results on the optimization and generalization of the attention network. First, the neural network function after every gradient descent iteration admits a decomposition into a function with bounded Reproducing Kernel Hilbert Space (RKHS) norm and a uniformly small residual. Second, local Rademacher complexity converts this decomposition into a regression risk bound. Kernel gradient descent on the same ball attains the same rate, while our theorem gives explicit sufficient width and auxiliary-sample conditions for simultaneous network training. Our analysis is further extended to covariates supported on RR^d through a novel normalization method. Under a sub-Gaussian covariate distribution, the attention network achieves a nearly optimal regression risk cO((log n)n^−s'(d+3)/(s'(d+3)+d)) without a density assumption.
ER -