Memory by Design: Bayesian Layers for Sequence Modeling
Abstract
We introduce the \emph{design-model framework}: a way to derive efficient recurrent sequence maps from explicit assumptions about memory. A design model writes evidence into memory by exact Bayesian filtering; a query- dependent readout produces a predictive distribution whose mean is the layer output. In our linear-Gaussian instantiation, the \emph{Bayesian Layer} propagates both a mean and a covariance: the covariance tracks uncertainty over stored associations, steering writes toward uncertain directions, attenuating gains as evidence accumulates, and preserving confident memories. The same framework unifies several sub-quadratic recurrences: linear attention, GLA, and Mamba-2/SSD are exact filters under a latent-input design model, whereas DeltaNet and related Delta-rule models are covariance-reset reductions of the Bayesian Layer's design model. Restoring covariance propagation yields closed-form predictions for retrieval dynamics, which we verify empirically, and improves robustness beyond the training regime in controlled collision studies, learned associative recall, and the Zoology MQAR benchmark. Training from scratch on WikiText-103 under matched state budgets lowers perplexity on associative-recall hits. Distilling Bayesian Layers into a pretrained 340M Gated DeltaNet improves RULER long-context retrieval over a matched-compute control, at a 2.5--2.7\% held-out perplexity cost.
Related
- Generative sequence modeling for infinite memory processes via predictive states
- Variational Filtering for Bayesian Last Layers with an Independent Binary Probit Surrogate
- Transfer learning in bayesian optimization for aircraft design
- Modern Bayesian Modeling and Inference
- Kumaraswamy-Teissier Distribution: Bayesian and Non-Bayesian Inference, Regression Modeling and Climate Application
BibTeX
@misc{dowling2026memory,
title = {{Memory by Design: Bayesian Layers for Sequence Modeling}},
author = {Matthew Dowling and Hyungju Jeon and Cristina Savin and Il Memming Park},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.31163},
eprint = {2605.31163},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.31163},
}
RIS
TY - UNPB
TI - Memory by Design: Bayesian Layers for Sequence Modeling
AU - Matthew Dowling
AU - Hyungju Jeon
AU - Cristina Savin
AU - Il Memming Park
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2605.31163
UR - https://arxiv.org/abs/2605.31163
AB - We introduce the \emph{design-model framework}: a way to derive efficient recurrent sequence maps from explicit assumptions about memory. A design model writes evidence into memory by exact Bayesian filtering; a query- dependent readout produces a predictive distribution whose mean is the layer output. In our linear-Gaussian instantiation, the \emph{Bayesian Layer} propagates both a mean and a covariance: the covariance tracks uncertainty over stored associations, steering writes toward uncertain directions, attenuating gains as evidence accumulates, and preserving confident memories. The same framework unifies several sub-quadratic recurrences: linear attention, GLA, and Mamba-2/SSD are exact filters under a latent-input design model, whereas DeltaNet and related Delta-rule models are covariance-reset reductions of the Bayesian Layer's design model. Restoring covariance propagation yields closed-form predictions for retrieval dynamics, which we verify empirically, and improves robustness beyond the training regime in controlled collision studies, learned associative recall, and the Zoology MQAR benchmark. Training from scratch on WikiText-103 under matched state budgets lowers perplexity on associative-recall hits. Distilling Bayesian Layers into a pretrained 340M Gated DeltaNet improves RULER long-context retrieval over a matched-compute control, at a 2.5--2.7\% held-out perplexity cost.
ER -