The Geometry of Limits in ReLU Function Spaces
Abstract
We study the geometry and closure of spaces of functions realized by ReLU neural networks of fixed architecture. We first establish a general result about limits of continuous piecewise-linear (CPWL) functions: the minimum number of convex sets on which a function is affine is lower semicontinuous under pointwise convergence. The proof uses a connection to hypergraph coloring and a compactness principle, and yields quantitative bounds on the polyhedral complexity of continuous limits. We then characterize the functions realizable by shallow ReLU networks with vector-valued outputs, extending a known scalar-valued characterization. Using this and the geometry of limiting hyperplane arrangements, we prove that fixed-width shallow scalar ReLU realization spaces are closed in the weighted supremum norm on the whole input space. In contrast, we construct an explicit sequence of networks with architecture (2,3,2|1) whose limit is not realizable by that architecture. Non-closedness holds both globally in weighted supremum norm and under uniform convergence on compact sets with nonempty interior, disproving a conjecture of Petersen, Raslan, and Voigtlaender. Finally, we show that non-closedness for a fixed scalar-output ReLU architecture necessarily involves a \emph{blow-up residue}, in which diverging output coefficients amplify an emerging linear dependence among hidden features. Together, these results provide geometric and functional-analytic tools for studying the boundaries and degenerations of ReLU realization spaces.
Related
- Sharp embeddings between quasi-Banach Besov spaces and shallow ReLU variation spaces
- Sharp Sobolev Sandwich and Approximation Rates of Radon-Domain Lp Ridge Integral Spaces for ReLUk Networks
- Schauder and Riesz Bases for Higher-Dimensional Lebesgue Spaces Inspired by ReLU Neural Networks
- Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas
- Boundary Geometry Determines Optimal Shallow ReLU Approximation under Weighted Coefficient Constraints
BibTeX
@misc{lindsey2026geometry,
title = {{The Geometry of Limits in ReLU Function Spaces}},
author = {Kathryn A. Lindsey},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33779},
eprint = {2610.33779},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33779},
}
RIS
TY - UNPB
TI - The Geometry of Limits in ReLU Function Spaces
AU - Kathryn A. Lindsey
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2610.33779
UR - https://arxiv.org/abs/2610.33779
AB - We study the geometry and closure of spaces of functions realized by ReLU neural networks of fixed architecture. We first establish a general result about limits of continuous piecewise-linear (CPWL) functions: the minimum number of convex sets on which a function is affine is lower semicontinuous under pointwise convergence. The proof uses a connection to hypergraph coloring and a compactness principle, and yields quantitative bounds on the polyhedral complexity of continuous limits. We then characterize the functions realizable by shallow ReLU networks with vector-valued outputs, extending a known scalar-valued characterization. Using this and the geometry of limiting hyperplane arrangements, we prove that fixed-width shallow scalar ReLU realization spaces are closed in the weighted supremum norm on the whole input space. In contrast, we construct an explicit sequence of networks with architecture (2,3,2|1) whose limit is not realizable by that architecture. Non-closedness holds both globally in weighted supremum norm and under uniform convergence on compact sets with nonempty interior, disproving a conjecture of Petersen, Raslan, and Voigtlaender. Finally, we show that non-closedness for a fixed scalar-output ReLU architecture necessarily involves a \emph{blow-up residue}, in which diverging output coefficients amplify an emerging linear dependence among hidden features. Together, these results provide geometric and functional-analytic tools for studying the boundaries and degenerations of ReLU realization spaces.
ER -