Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence
Abstract
Deep neural networks approximate functions by composing affine maps with nonlinear activations, but how composition itself creates approximation power is not yet fully understood. We investigate a fundamental mechanism: geometrically weighted sums of iterates of a single scalar generator function. This mechanism underpins the classical tent-map construction of the function x − x2 and related recursive representations used by Yarotsky, W. E, et al., to analyze the approximation powers of deep neural networks. First, we establish a rigidity theorem: for continuous piecewise linear generators with a finite number of segments, any C3 function that can be represented in this way is at most quadratic. For non-affine quadratic functions, the geometric factor is at least 1/4. This result both reveals limitations of the tent-map approach and complements existing methods based on hierarchical bases and recursive polynomial constructions. Second, using an exact remainder identity as guidance, we construct a smooth generator whose iterates yield doubly exponential error decay in total depth for square approximation and, through multiplication modules, for each fixed polynomial. For power series with absolutely summable coefficients on [−1,1]d, distributing depth according to monomial degree yields a uniform approximation error of order O(e−cL1/d) on each interior cube. These findings demonstrate how generator dynamics and remainder estimates govern depth allocation and approximation rates of deep neural networks.
Related
- Convergence of repeated-pole rational approximation of the exponential on infinite sectors
- A Rigidity Property of the Deformed q-Exponential Function
- Entropy instability and rigidity for the exponential semilinear heat equation
- Exponential Convergence of the Sinkhorn Algorithm for the Schrödinger Bridge with Regime Switching
- Quantitative Centre Rigidity under Riemannian Approximation in Heisenberg Groups
BibTeX
@misc{huang2026neural,
title = {{Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence}},
author = {Wentao Huang and Haizhang Zhang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.25874},
eprint = {2609.25874},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.25874},
}
RIS
TY - UNPB
TI - Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence
AU - Wentao Huang
AU - Haizhang Zhang
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2609.25874
UR - https://arxiv.org/abs/2609.25874
AB - Deep neural networks approximate functions by composing affine maps with nonlinear activations, but how composition itself creates approximation power is not yet fully understood. We investigate a fundamental mechanism: geometrically weighted sums of iterates of a single scalar generator function. This mechanism underpins the classical tent-map construction of the function x − x^2 and related recursive representations used by Yarotsky, W. E, et al., to analyze the approximation powers of deep neural networks. First, we establish a rigidity theorem: for continuous piecewise linear generators with a finite number of segments, any C^3 function that can be represented in this way is at most quadratic. For non-affine quadratic functions, the geometric factor is at least 1/4. This result both reveals limitations of the tent-map approach and complements existing methods based on hierarchical bases and recursive polynomial constructions. Second, using an exact remainder identity as guidance, we construct a smooth generator whose iterates yield doubly exponential error decay in total depth for square approximation and, through multiplication modules, for each fixed polynomial. For power series with absolutely summable coefficients on [−1,1]^d, distributing depth according to monomial degree yields a uniform approximation error of order O(e^−cL^1/d) on each interior cube. These findings demonstrate how generator dynamics and remainder estimates govern depth allocation and approximation rates of deep neural networks.
ER -