Optimal approximation of smooth functions by ReLU neural networks
Abstract
We study uniform approximation of the Hölder ball ℋ1s([0,1]D) by fully connected ReLU networks. Except in the one-dimensional Lipschitz case, existing constructions remain separated from the information-theoretic efficiency limit. This article closes the gap for every s>0 and D≥1: networks with width O(N), depth O(L), and parameter magnitude O(1) attain uniform approximation error of order (N2L2log N)−s/D. No relation between width and depth is required. The construction uses new encoding and decoding mechanisms that exploit higher-order finite-difference bounds for local Taylor coefficients of the target function. As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate n−2s/(2s+D) without additional logarithmic factors. The paper also gives a detailed exposition of the key approximation techniques, covering both established and new mechanisms.
Related
- Fully Deterministic Shallow Neural Networks with Localized Activations. Part I: Construction and Optimal Sobolev Approximation
- On the Optimal Linear Minimization Oracle Complexity of Active-Hull Methods for Minimizing Convex Hölder Smooth Functions
- Deep ReLU Networks Can Fail the Best Approximation Property for Continuous Target Functions
- ReLU-spline activation functions for Neural Network approximation of solutions to differential equations
- Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
BibTeX
@misc{nagler2026optimal,
title = {{Optimal approximation of smooth functions by ReLU neural networks}},
author = {Thomas Nagler},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22693},
eprint = {2610.22693},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22693},
}
RIS
TY - UNPB
TI - Optimal approximation of smooth functions by ReLU neural networks
AU - Thomas Nagler
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.22693
UR - https://arxiv.org/abs/2610.22693
AB - We study uniform approximation of the Hölder ball ℋ_1^s([0,1]^D) by fully connected ReLU networks. Except in the one-dimensional Lipschitz case, existing constructions remain separated from the information-theoretic efficiency limit. This article closes the gap for every s>0 and D≥1: networks with width O(N), depth O(L), and parameter magnitude O(1) attain uniform approximation error of order (N^2L^2log N)^−s/D. No relation between width and depth is required. The construction uses new encoding and decoding mechanisms that exploit higher-order finite-difference bounds for local Taylor coefficients of the target function. As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate n^−2s/(2s+D) without additional logarithmic factors. The paper also gives a detailed exposition of the key approximation techniques, covering both established and new mechanisms.
ER -