Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
Abstract
Cortical neurons are complex, multi-timescale processors wired into recurrent circuits, shaped by long evolutionary pressure under stringent biological constraints. Mainstream machine learning, by contrast, predominantly builds models from extremely simple units, a default inherited from early neural-network theory. We treat this as a normative architectural question. How should one split a fixed parameter budget P between the number of units N, per-unit effective complexity ke, and per-unit connectivity kc? What controls the optimal allocation? This calls for a model in which per-unit complexity can be tuned independently of width and connectivity. Accordingly, we introduce the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons, chosen to mirror functional components of cortical neurons. The architecture allows for individually adjusting N, ke, and kc and trains stably across orders of magnitude in scale. We evaluate the model on two qualitatively different sequence benchmarks: the neuromorphic SHD-Adding task and Enwik8 character-level language modeling. Performance improves monotonically along each of the three axes individually. Under a fixed budget, a clear non-trivial optimum emerges in their tradeoff, and larger budgets favor both more numerous \emph{and} more complex neurons. A closed-form information-theoretic proxy captures these tradeoffs and attributes the diminishing returns at two ends to: per-neuron signal-to-noise saturation and across-neuron redundancy. A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the proxy. This suggests that the simple-unit default in ML is not obviously optimal once this tradeoff surface is probed, and offers a normative lens on cortex's reliance on complex spatio-temporal integrators.
Related
- Recurrent Neural Networks and (Time-Uniform) Universal Approximation of Bayesian Filters
- A Polynomial-Scaling PDE Solver with Entanglement-Basis Tensor Networks
- Doubly-robust causal inference for multi-valued treatment regimes with recurrent neural networks
- Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
- On architectural choices for interpretability and thermodynamic consistency in Physically Recurrent Neural Networks in the low-data regime
BibTeX
@misc{spieler2026scaling,
title = {{Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons}},
author = {Aaron Spieler and Georg Martius and Anna Levina},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.12049},
eprint = {2605.12049},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.12049},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
AU - Aaron Spieler
AU - Georg Martius
AU - Anna Levina
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2605.12049
UR - https://arxiv.org/abs/2605.12049
AB - Cortical neurons are complex, multi-timescale processors wired into recurrent circuits, shaped by long evolutionary pressure under stringent biological constraints. Mainstream machine learning, by contrast, predominantly builds models from extremely simple units, a default inherited from early neural-network theory. We treat this as a normative architectural question. How should one split a fixed parameter budget P between the number of units N, per-unit effective complexity k_e, and per-unit connectivity k_c? What controls the optimal allocation? This calls for a model in which per-unit complexity can be tuned independently of width and connectivity. Accordingly, we introduce the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons, chosen to mirror functional components of cortical neurons. The architecture allows for individually adjusting N, k_e, and k_c and trains stably across orders of magnitude in scale. We evaluate the model on two qualitatively different sequence benchmarks: the neuromorphic SHD-Adding task and Enwik8 character-level language modeling. Performance improves monotonically along each of the three axes individually. Under a fixed budget, a clear non-trivial optimum emerges in their tradeoff, and larger budgets favor both more numerous \emph{and} more complex neurons. A closed-form information-theoretic proxy captures these tradeoffs and attributes the diminishing returns at two ends to: per-neuron signal-to-noise saturation and across-neuron redundancy. A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the proxy. This suggests that the simple-unit default in ML is not obviously optimal once this tradeoff surface is probed, and offers a normative lens on cortex's reliance on complex spatio-temporal integrators.
ER -