Fast Weight Attention for Continual Learning
Abstract
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (xt,yt)=(ϕ(kt−1),vt). The common same-step association (ϕ(kt),vt) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.
Related
- A fast risk assessment method for power balance under extreme weather conditions based on deep reinforcement learning with a coupled attention mechanism
- Reliable Replay through Spatial Coherence in Online Continual Learning
- Generative Replay for Equilibrium Propagation: Toward Edge-Friendly Continual Learning in Hopfield Networks
- ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs
- A Dynamical Theory of LoRA in Continual Learning
BibTeX
@misc{zhang2026fast,
title = {{Fast Weight Attention for Continual Learning}},
author = {Yifan Zhang and Steve Ta and Jasper Zhang and Jichen Feng and Shuzhen Li and Yongxin Zhang and Yifeng Liu and Huizhuo Yuan and Mengdi Wang and Quanquan Gu and Andrew Chi-Chih Yao},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.27763},
eprint = {2608.27763},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.27763},
}
RIS
TY - UNPB
TI - Fast Weight Attention for Continual Learning
AU - Yifan Zhang
AU - Steve Ta
AU - Jasper Zhang
AU - Jichen Feng
AU - Shuzhen Li
AU - Yongxin Zhang
AU - Yifeng Liu
AU - Huizhuo Yuan
AU - Mengdi Wang
AU - Quanquan Gu
AU - Andrew Chi-Chih Yao
PY - 2026
DA - 2026/08/27
PB - arXiv
DO - 10.48550/arxiv.2608.27763
UR - https://arxiv.org/abs/2608.27763
AB - Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(ϕ(k_t−1),v_t). The common same-step association (ϕ(k_t),v_t) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.
ER -