Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory
Abstract
Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the $0.10$ chance floor; adding binding raises it to $0.92$. Trained only on 32 kb genomic contexts, DT reaches $0.927$ AUROC at 131 kb and $0.856$ near 1 Mb, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.
Related
- TransBind2: Improving Transcription Factor-DNA Binding Prediction with Multimodal Data and Bidirectional Cross Attention
- Antibodies with Engineered Fc Domains Having Absolute Binding Selectivity to either FcγRIIa or FcγRI Delineate the Respective Effector Phenotypes by Human Monocytes and Macrophages
- A Context-Aware 14-3-3 Binding Predictor Enabled by Data Augmentation, Multi-Scale Biological Features, and Protein Language Model Embeddings
- CaliPPer: quantifying, predicting and improving AI model performance for binding prediction
- Reducing dietary acid-binding capacity of low-zinc diets increased growth performance of weaned pigs in a commercial environment.
BibTeX
@misc{raghavan2026doubletake,
title = {{Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory}},
author = {Vakul Raghavan and Utkarsh Grover and Mingyang Mao and Chengwen Wang and Xiaomin Lin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13073},
eprint = {2610.13073},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13073},
}
RIS
TY - UNPB
TI - Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory
AU - Vakul Raghavan
AU - Utkarsh Grover
AU - Mingyang Mao
AU - Chengwen Wang
AU - Xiaomin Lin
PY - 2026
DA - 2026/10/18
PB - arXiv
DO - 10.48550/arxiv.2610.13073
UR - https://arxiv.org/abs/2610.13073
AB - Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the $0.10$ chance floor; adding binding raises it to $0.92$. Trained only on 32 kb genomic contexts, DT reaches $0.927$ AUROC at 131 kb and $0.856$ near 1 Mb, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.
ER -