Express Language Modeling
Abstract
We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering log3/2(n)/s approximation error with only O(s) memory and O(s2 log2(n)) compression overhead for a sequence of length n. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding.
Related
- Language Modeling is Monotone Compression
- Values of Absorbing Recursive Games Express All Real Algebraic Numbers
- On the language of palindromic reduced words in a Coxeter group
- Estimating Great Expectations Under Autoregressive Language Models with Potentials
- A Distributional Framework for Behavioral Characterization and Risk Management of Large Language Models
BibTeX
@misc{gong2026express,
title = {{Express Language Modeling}},
author = {Albert Gong and Annabelle Michael Carrell and Raaz Dwivedi and Lester Mackey},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.10944},
eprint = {2606.10944},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.10944},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Express Language Modeling
AU - Albert Gong
AU - Annabelle Michael Carrell
AU - Raaz Dwivedi
AU - Lester Mackey
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2606.10944
UR - https://arxiv.org/abs/2606.10944
AB - We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering log^3/2(n)/s approximation error with only O(s) memory and O(s^2 log^2(n)) compression overhead for a sequence of length n. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding.
ER -