Interdomain Attention: Beyond Token-Level Key-Value Memory
Abstract
Softmax attention-based transformers and recurrent memory-based models (e.g., state space models (SSMs)) form the foundation of modern sequence models. Hybrid architectures aim to combine the expressivity from attention and the computational efficiency from recurrent memory, yet so far common practices simply stack both attention and recurrent layers. We propose Interdomain Attention (IA) as a novel hybrid attention architecture that mixes attention and recurrent memory in the same layer. Motivated by a kernel regression view of softmax attention, IA compresses values and kernel features of keys into a recurrent memory parameterized by an SSM. Then IA approximates dot-product attention by implicitly reconstructing the attention matrix and values from memory states; importantly this construction maintains SSM's computational efficiency advantage. We demonstrate IA's efficacy by designing and pre-training IA network architectures, where ablation studies identify attention-like readout as the primary source of gain over SSMs. Across 125M-1.3B models pre-trained on text data, IA improves over an S4D baseline at matched memory size. It also outperforms softmax attention-based Transformer in perplexity and accuracy on common-sense reasoning tasks. Based on Caduceus backbone, IA matches or outperforms the original SSM-based architecture in both perplexity and downstream tasks on genomic data.
Related
- Token-Level Attribution of Multi-Mode LLM Watermarks: Optimal Rule and Belief Weighting
- Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
- Effective Dissipation and Asymptotic Stability in a Finite-Level Quantum System with Wave-Induced Memory
- Beyond Multiplicity Correction: Sharp Retrieval Thresholds for Attention with Correlated Distractors
- Beyond Source-Level Transfer: Sample-Level Learning for High-Dimensional Quantile Regression
BibTeX
@misc{kiyohara2026interdomain,
title = {{Interdomain Attention: Beyond Token-Level Key-Value Memory}},
author = {Naoki Kiyohara and Harrison Bo Hua Zhu and Riccardo El Hassanin and Zhuo Sun and Wenlong Chen and Samir Bhatt and Yingzhen Li},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.24330},
eprint = {2605.24330},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.24330},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Interdomain Attention: Beyond Token-Level Key-Value Memory
AU - Naoki Kiyohara
AU - Harrison Bo Hua Zhu
AU - Riccardo El Hassanin
AU - Zhuo Sun
AU - Wenlong Chen
AU - Samir Bhatt
AU - Yingzhen Li
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2605.24330
UR - https://arxiv.org/abs/2605.24330
AB - Softmax attention-based transformers and recurrent memory-based models (e.g., state space models (SSMs)) form the foundation of modern sequence models. Hybrid architectures aim to combine the expressivity from attention and the computational efficiency from recurrent memory, yet so far common practices simply stack both attention and recurrent layers. We propose Interdomain Attention (IA) as a novel hybrid attention architecture that mixes attention and recurrent memory in the same layer. Motivated by a kernel regression view of softmax attention, IA compresses values and kernel features of keys into a recurrent memory parameterized by an SSM. Then IA approximates dot-product attention by implicitly reconstructing the attention matrix and values from memory states; importantly this construction maintains SSM's computational efficiency advantage. We demonstrate IA's efficacy by designing and pre-training IA network architectures, where ablation studies identify attention-like readout as the primary source of gain over SSMs. Across 125M-1.3B models pre-trained on text data, IA improves over an S4D baseline at matched memory size. It also outperforms softmax attention-based Transformer in perplexity and accuracy on common-sense reasoning tasks. Based on Caduceus backbone, IA matches or outperforms the original SSM-based architecture in both perplexity and downstream tasks on genomic data.
ER -