Learning infinite context windows in recurrent architectures via spatial neural computing
Abstract
Recurrent neural networks (RNNs) offer linear-time scaling with sequence length while requiring only constant memory, yet they struggle to capture long-range dependencies due to vanishing gradients and limited receptive fields. To address these limitations, we introduce a second-order recurrent model in which the standard neuron-to-neuron communication is replaced by a spatially evolving field governed by (discretized) partial differential equations. Drawing inspiration from the role of cortical waves in brain computation, this mechanism allows structured spatiotemporal patterns to serve as an implicit, high-capacity memory. We show that the resulting model is equivalent to a structured infinite-order RNN in which the current state depends explicitly on its entire history of past states, yielding an effectively unbounded receptive field with a fixed number of parameters. We further derive constructive conditions to ensure marginal stability, constraining the gradient spectrum on the unit circle and thereby eliminating vanishing and exploding gradients. Empirically, the proposed architecture outperforms other recurrent models on long-horizon benchmarks while using substantially fewer parameters, demonstrating that spatial dynamics can effectively bridge the gap between efficient inference and long-term memory.
Related
- Deterministic DNA-first discovery reveals recurrent genomic architectures
- Population-scale Y chromosome assemblies reveal recurrent remodeling within constrained architectures
- Evaluating in-context learning with prompting regimes for transformer-based synthetic health tabular data generation
- Learning to play with spikes: characterizing, predicting, and engineering unsupervised plasticity rules for spiking reservoir computing
- A theory of noisy linear recurrent neural network learning dynamics
BibTeX
@misc{salvadorpomarol2026learning,
title = {{Learning infinite context windows in recurrent architectures via spatial neural computing}},
author = {Aleix Salvador-Pomarol and Arthur N. Montanari and Earl K. Miller and Adilson E. Motter and Jorge Cortés},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10690},
eprint = {2610.10690},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10690},
}
RIS
TY - UNPB
TI - Learning infinite context windows in recurrent architectures via spatial neural computing
AU - Aleix Salvador-Pomarol
AU - Arthur N. Montanari
AU - Earl K. Miller
AU - Adilson E. Motter
AU - Jorge Cortés
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10690
UR - https://arxiv.org/abs/2610.10690
AB - Recurrent neural networks (RNNs) offer linear-time scaling with sequence length while requiring only constant memory, yet they struggle to capture long-range dependencies due to vanishing gradients and limited receptive fields. To address these limitations, we introduce a second-order recurrent model in which the standard neuron-to-neuron communication is replaced by a spatially evolving field governed by (discretized) partial differential equations. Drawing inspiration from the role of cortical waves in brain computation, this mechanism allows structured spatiotemporal patterns to serve as an implicit, high-capacity memory. We show that the resulting model is equivalent to a structured infinite-order RNN in which the current state depends explicitly on its entire history of past states, yielding an effectively unbounded receptive field with a fixed number of parameters. We further derive constructive conditions to ensure marginal stability, constraining the gradient spectrum on the unit circle and thereby eliminating vanishing and exploding gradients. Empirically, the proposed architecture outperforms other recurrent models on long-horizon benchmarks while using substantially fewer parameters, demonstrating that spatial dynamics can effectively bridge the gap between efficient inference and long-term memory.
ER -