Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations
Abstract
Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture Φ with PΦ parameter slots, we write θf=𝒢(ξf), where 𝒢:ℝM→ℝPΦ is a parameter generator and ξf∈ℝM is a latent representation of the target function f. The architecture Φ and the generator 𝒢 are shared across the entire target class, while each target f is represented by its own latent vector ξf, with Φ𝒢(ξf) approximating f. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension M and the network budget P is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures Φ satisfying PΦ≤ P and affine generators 𝒢:ℝM→ ℝPΦ, we prove that the optimal worst-case uniform approximation error over the unit ball of α-Hölder functions on [0,1]d, where 0<α≤1, has the sharp order $ \bigl(P\min\{M,P\}\bigr)^{-α/d}. $ In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.
Related
- Approximation of solutions of parameter-dependent problems by residual neural networks
- Lp approximation results for infinite dimensional Neural Networks
- Recurrent Neural Networks and (Time-Uniform) Universal Approximation of Bayesian Filters
- Optimal approximation of smooth functions by ReLU neural networks
- Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence
BibTeX
@misc{zhang2026sharp,
title = {{Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations}},
author = {Shijun Zhang},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.31157},
eprint = {2608.31157},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.31157},
}
RIS
TY - UNPB
TI - Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations
AU - Shijun Zhang
PY - 2026
DA - 2026/08/31
PB - arXiv
DO - 10.48550/arxiv.2608.31157
UR - https://arxiv.org/abs/2608.31157
AB - Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture Φ with P_Φ parameter slots, we write θ_f=𝒢(ξ_f), where 𝒢:ℝ^M→ℝ^P_Φ is a parameter generator and ξ_f∈ℝ^M is a latent representation of the target function f. The architecture Φ and the generator 𝒢 are shared across the entire target class, while each target f is represented by its own latent vector ξ_f, with Φ_𝒢(ξ_f) approximating f. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension M and the network budget P is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures Φ satisfying P_Φ≤ P and affine generators 𝒢:ℝ^M→ ℝ^P_Φ, we prove that the optimal worst-case uniform approximation error over the unit ball of α-Hölder functions on [0,1]^d, where 0<α≤1, has the sharp order $ \bigl(P\min\{M,P\}\bigr)^{-α/d}. $ In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.
ER -