On the Architectural Complexity of Neural Networks
Abstract
We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs). This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations -- lower level information that is often abstracted. Our framework enables two novel objectives: (1) analysis of the evolution of architectural complexity over deep learning history, and (2) automatic construction of novel architectures based on new types of tensor operations. Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity. Moreover, we identify several large classes of higher complexity architectures that have not yet been explored. We then collect a dataset of 3,000+ higher complexity architectures, which we publicly release at: https://github.com/combinatoriallabs/ArchitecturalComplexity.
Related
- On architectural choices for interpretability and thermodynamic consistency in Physically Recurrent Neural Networks in the low-data regime
- Complexity of Reachability and Verification Problems in Neural Networks
- Oblique Projection Networks (OPN) Reference guide - variants, code, tests, complexity, and the application scope of oblique projections
- Logical Neural Belief Propagation for Linear-Complexity Decoding of Surface Codes
- Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
BibTeX
@misc{cooper2026architectural,
title = {{On the Architectural Complexity of Neural Networks}},
author = {Nicholas J. Cooper and François G. Meyer and Michael L. Roberts and Carlos Zapata-Carratalá and Lijun Chen and Danna Gurari},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.04325},
eprint = {2605.04325},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.04325},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - On the Architectural Complexity of Neural Networks
AU - Nicholas J. Cooper
AU - François G. Meyer
AU - Michael L. Roberts
AU - Carlos Zapata-Carratalá
AU - Lijun Chen
AU - Danna Gurari
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2605.04325
UR - https://arxiv.org/abs/2605.04325
AB - We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs). This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations -- lower level information that is often abstracted. Our framework enables two novel objectives: (1) analysis of the evolution of architectural complexity over deep learning history, and (2) automatic construction of novel architectures based on new types of tensor operations. Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity. Moreover, we identify several large classes of higher complexity architectures that have not yet been explored. We then collect a dataset of 3,000+ higher complexity architectures, which we publicly release at: https://github.com/combinatoriallabs/ArchitecturalComplexity.
ER -