PreprintCC BY 4.0arXiv
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
Abstract
While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that rerouting a selective minority of sub-blocks to high precision is sufficient to recover the baseline model performance.
Related
- FlashAttention-Compatible Modality-Mutual Attention with Virtual Reordering
- Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
- Learned Look-Ahead Splitting Rule for CART
- Day-Ahead Electricity Price Forecasting Using a Multivariate Group Lasso Method
- Real-Time 24-hour Ahead Forecasting of Solar Energetic Particle Events with SEPNET-PRISM-R
BibTeX
@misc{budzinskiy2026lampattention,
title = {{LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators}},
author = {Stanislav Budzinskiy and Marian Gloser and Tolunay Yilmaz and Ying Hong Tham and Yuanyi Lin and Wenyi Fang and Fan Wu and Philipp Petersen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.39361},
eprint = {2609.39361},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.39361},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
AU - Stanislav Budzinskiy
AU - Marian Gloser
AU - Tolunay Yilmaz
AU - Ying Hong Tham
AU - Yuanyi Lin
AU - Wenyi Fang
AU - Fan Wu
AU - Philipp Petersen
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.39361
UR - https://arxiv.org/abs/2609.39361
AB - While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that rerouting a selective minority of sub-blocks to high precision is sufficient to recover the baseline model performance.
ER -