Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

Abstract

We develop a general framework for analyzing representation costs induced by parameter-space regularizers in data-fitting methods. For an arbitrary parametric method, we define its representation cost and native function space, prove existence, and identify conditions under which parameter-space and function-space problems have equal infimal values and minimizers transfer between them. This framework yields representer theorems and recovers classical formulations---including kernel methods and RKHSs, wavelets and Besov spaces, and shallow neural networks and variation spaces---as special cases. Our main new results concern depth-L feedforward ReLU networks with weight-decay regularization. For these networks, we prove that the representation cost is a power of a quasi-seminorm and that, under suitable hypotheses, the native space is a quasi-Banach space with nonconvex unit ball when L > 2. These results identify a novel depth-dependent quasi-Banach geometry induced by weight decay.

Empirical Bernstein in smooth Banach spaces
Diego Martinez-Taboada, Aaditya Ramdas
The Annals of Applied Probability1 Jun 2026Peer-reviewedLicence not statedFull text16 citedStatisticsOpenAlex
On the foundations of logic and probability, II. Modal probability spaces
Loic Cosyns
arXiv20 Oct 2026PreprintarXiv licenceFull textLogic & foundations ProbabilityarXiv
Uniform Embeddings of Banach Space Spheres into Hilbert Spaces and Property (H)
Qingjin Cheng, Yue Wang, Yipeng Zhang
arXiv5 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv
Improved Bounds For The Lipschitz Retraction Constant In Classical Banach Spaces
Shriram Mahesh
arXiv11 Sep 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv
Compactness in Combinatorial Banach Spaces and Beyond
Jacek Gulgowski, Paweł Sawicki
arXiv7 Oct 2026PreprintCC BY 4.0Full textAnalysis & PDEsarXiv

Cite

BibTeX RIS

BibTeX

@misc{ongie2026representation,
  title = {{Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks}},
  author = {Greg Ongie and Rahul Parhi},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2606.14954},
  eprint = {2606.14954},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.14954},
}

RIS

TY  - UNPB
TI  - Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks
AU  - Greg Ongie
AU  - Rahul Parhi
PY  - 2026
DA  - 2026/09/05
PB  - arXiv
DO  - 10.48550/arxiv.2606.14954
UR  - https://arxiv.org/abs/2606.14954
AB  - We develop a general framework for analyzing representation costs induced by parameter-space regularizers in data-fitting methods. For an arbitrary parametric method, we define its representation cost and native function space, prove existence, and identify conditions under which parameter-space and function-space problems have equal infimal values and minimizers transfer between them. This framework yields representer theorems and recovers classical formulations---including kernel methods and RKHSs, wavelets and Besov spaces, and shallow neural networks and variation spaces---as special cases. Our main new results concern depth-L feedforward ReLU networks with weight-decay regularization. For these networks, we prove that the representation cost is a power of a quasi-seminorm and that, under suitable hypotheses, the native space is a quasi-Banach space with nonconvex unit ball when L > 2. These results identify a novel depth-dependent quasi-Banach geometry induced by weight decay.
ER  -

Abstract

Related

Cite