{
  "abstract": "Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step $t$ is the prefix-aligned pair $(\\mathbf{x}_t,\\mathbf{y}_t)=(ϕ(\\mathbf{k}_{t-1}),\\mathbf{v}_t)$. The common same-step association $(ϕ(\\mathbf{k}_t),\\mathbf{v}_t)$ remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2608.27763",
  "author_total": 11,
  "authors": [
    {
      "name": "Yifan Zhang"
    },
    {
      "name": "Steve Ta"
    },
    {
      "name": "Jasper Zhang"
    },
    {
      "name": "Jichen Feng"
    },
    {
      "name": "Shuzhen Li"
    },
    {
      "name": "Yongxin Zhang"
    },
    {
      "name": "Yifeng Liu"
    },
    {
      "name": "Huizhuo Yuan"
    },
    {
      "name": "Mengdi Wang"
    },
    {
      "name": "Quanquan Gu"
    },
    {
      "name": "Andrew Chi-Chih Yao"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/fsp35syw5kn29.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2608.27763",
  "fulltext_url": null,
  "id": "fsp35syw5kn29",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2608.27763",
  "language": null,
  "licence": "LicenseRef-arXiv-nonexclusive-1.0",
  "licence_label": "arXiv licence",
  "licence_source": "article",
  "licence_url": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.CL"
    ],
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "stat.ML"
    ],
    [
      "openalex",
      "1702|Topic Modeling"
    ]
  ],
  "openalex": "W7204828645",
  "pdf_url": "https://arxiv.org/pdf/2608.27763",
  "pmcid": null,
  "pmid": null,
  "published": "2026-08-27",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/fsp35syw5kn29.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:34Z",
      "native_id": "2608.27763",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2608.27763"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ML",
      "subfield": "statistics"
    }
  ],
  "title": "Fast Weight Attention for Continual Learning",
  "title_text": "Fast Weight Attention for Continual Learning",
  "type": "preprint",
  "updated_at": "2026-10-30T12:17:34Z",
  "url": "https://research.draconitic.com/paper/fsp35syw5kn29",
  "venue": "arXiv",
  "versions": []
}