{
  "abstract": "We study nonparametric regression using an over-parameterized two-layer ReLU neural network with fixed covariance-based channel attention, trained by gradient descent with both layers updated simultaneously. The spherical covariates follow an arbitrary fixed non-atomic distribution, and the target function lies in an interpolation space with spectral bias widely studied in the statistical learning literature. We prove that early-stopped training achieves regression risk $\\cO(\\eps_n^2)$, where $\\eps_n$ is the critical population rate of the attention kernel $\\Kattn$. Its positive eigenvalues are $λ_j^{s'}$, where $λ_j$ are those of the ReLU feature covariance kernel $K$ and $s'=2s+3$ for $s\\ge1$. Under polynomial eigenvalue decay $λ_j\\asymp j^{-2α}$ with $α>1/2$, the regression risk bound $\\cO(\\eps_n^2)$ specializes to the minimax optimal rate $\\cO(n^{-2αs'/(2αs'+1)})$. Our analysis is based on two new technical results on the optimization and generalization of the attention network. First, the neural network function after every gradient descent iteration admits a decomposition into a function with bounded Reproducing Kernel Hilbert Space (RKHS) norm and a uniformly small residual. Second, local Rademacher complexity converts this decomposition into a regression risk bound. Kernel gradient descent on the same ball attains the same rate, while our theorem gives explicit sufficient width and auxiliary-sample conditions for simultaneous network training. Our analysis is further extended to covariates supported on $\\RR^d$ through a novel normalization method. Under a sub-Gaussian covariate distribution, the attention network achieves a nearly optimal regression risk $\\cO((\\log n)n^{-s'(d+3)/(s'(d+3)+d)})$ without a density assumption.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2611.01642",
  "author_total": 2,
  "authors": [
    {
      "name": "Yingzhen Yang"
    },
    {
      "name": "Ping Li"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/772zxf3e8dtw7.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2611.01642",
  "fulltext_url": null,
  "id": "772zxf3e8dtw7",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2611.01642",
  "language": null,
  "licence": "LicenseRef-arXiv-nonexclusive-1.0",
  "licence_label": "arXiv licence",
  "licence_source": "article",
  "licence_url": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "stat.ML"
    ]
  ],
  "openalex": null,
  "pdf_url": "https://arxiv.org/pdf/2611.01642",
  "pmcid": null,
  "pmid": null,
  "published": "2026-11-01",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/772zxf3e8dtw7.ris",
  "sources": [
    {
      "first_seen": "2026-11-03T12:21:21Z",
      "native_id": "2611.01642",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2611.01642"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ML",
      "subfield": "statistics"
    }
  ],
  "title": "Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention",
  "title_text": "Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention",
  "type": "preprint",
  "updated_at": "2026-11-05T15:20:35Z",
  "url": "https://research.draconitic.com/paper/772zxf3e8dtw7",
  "venue": "arXiv",
  "versions": []
}