Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression
Abstract
The statistical accuracy of neural networks depends on both their approximation power and the complexity of the class fitted from data. While increasing network size is a natural way to improve approximation, parameter magnitude provides another resource whose role must be quantified in both respects. We establish a sharp width--magnitude tradeoff at fixed depth using one elementary bounded $1$-Lipschitz Dyadic--Triangular Activation. For the unit β-Hölder ball on [0,1]d with 0<β≤1, the optimal Lp approximation error for 0<p<∞ is of order [N2log(eNT)]−β/d when the network width satisfies N≥2d+3 and the parameter magnitudes are bounded by T≥1. Matching lower bounds hold for every fixed globally Hölder activation; its Hölder exponent affects the constants but not the rate. Under bounded design densities and independent centered sub-Gaussian noise, approximate least squares over the full clipped class at depth $23$ attains the classical Hölder minimax risk 𝒪(M−2β/(2β+d)) without logarithmic loss whenever N2log(eNT)asymp Md/(2β+d), where M is the sample size. This yields a continuum of statistically optimal choices, ranging from unit parameter radius to fixed network size. At fixed size, four hidden layers with at most 8d+7 nonzero parameters give a near-optimal radius, while six layers with at most 8d+27 attain the optimal order log T=𝒪(η−d/β) at approximation error η. The same decoding method also yields fixed-size Transformer approximation.
Related
- Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation
- Diffusion Sensitivity Geometry and Strong Duality for Optimal Privacy-Utility Tradeoffs
- Optimal tradeoffs between variance and sensitivity, via Hellinger-Kantorovich geometry
- Asymptotic approximation of sensitivities in finite dimensional continuous data assimilation with application to parameter estimation
- Approximation of solutions of parameter-dependent problems by residual neural networks
BibTeX
@misc{li2026optimal,
title = {{Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression}},
author = {Baicheng Li and Zuowei Shen and Haizhao Yang and Shijun Zhang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.25710},
eprint = {2609.25710},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.25710},
}
RIS
TY - UNPB
TI - Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression
AU - Baicheng Li
AU - Zuowei Shen
AU - Haizhao Yang
AU - Shijun Zhang
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2609.25710
UR - https://arxiv.org/abs/2609.25710
AB - The statistical accuracy of neural networks depends on both their approximation power and the complexity of the class fitted from data. While increasing network size is a natural way to improve approximation, parameter magnitude provides another resource whose role must be quantified in both respects. We establish a sharp width--magnitude tradeoff at fixed depth using one elementary bounded $1$-Lipschitz Dyadic--Triangular Activation. For the unit β-Hölder ball on [0,1]^d with 0<β≤1, the optimal L^p approximation error for 0<p<∞ is of order [N^2log(eNT)]^−β/d when the network width satisfies N≥2d+3 and the parameter magnitudes are bounded by T≥1. Matching lower bounds hold for every fixed globally Hölder activation; its Hölder exponent affects the constants but not the rate. Under bounded design densities and independent centered sub-Gaussian noise, approximate least squares over the full clipped class at depth $23$ attains the classical Hölder minimax risk 𝒪(M^−2β/(2β+d)) without logarithmic loss whenever N^2log(eNT)asymp M^d/(2β+d), where M is the sample size. This yields a continuum of statistically optimal choices, ranging from unit parameter radius to fixed network size. At fixed size, four hidden layers with at most 8d+7 nonzero parameters give a near-optimal radius, while six layers with at most 8d+27 attain the optimal order log T=𝒪(η^−d/β) at approximation error η. The same decoding method also yields fixed-size Transformer approximation.
ER -