FlashAttention-Compatible Modality-Mutual Attention with Virtual Reordering
- 1ICT-Connected Composite Intelligence Research Center, Kyungpook National University, Daegu 41566, Republic of Korea
- 2Department of Artificial Intelligence, Kyungpook National University, Daegu 41566, Republic of Korea
Abstract
Multimodal large language models (MLLMs) process an interleaved image–text sequence with a single autoregressive backbone under causal attention. Since the causal mask prevents image tokens from attending to the question that follows them, their representations are computed without access to the query, which has been identified as a source of visual hallucination. Modality-mutual attention (MMA) mitigates this issue by unlocking the image-to-question region of the causal mask, and dual-order training (DOT) mitigates it by additionally training on the reversed text-first order. However, MMA turns the global causal mask into a sample-dependent dense mask. Consequently, attention cannot be executed via FlashAttention, which is the most efficient attention path, and falls back to a slower implementation. Furthermore, the positional information in the unlocked region does not match that of the reversed input order that motivates the unlock, that is, the positional semantics are inconsistent. To address both problems, we propose FlashAttention-compatible MMA with virtual reordering (FMMA-VR), which repairs the positional information of the unlocked region as if the two modalities were reordered, without physically reordering the sequence. The proposed method is implemented as a Triton-based region-aware FlashAttention-style kernel. On identically pretrained models, it accelerates the attention path by up to 3.59× over conventional MMA for long sequences during supervised fine-tuning (SFT). In post-SFT evaluations, it stays on par with conventional MMA across ten benchmark metrics under full fine-tuning. Under low-budget LoRA adaptation, where the positional shift of conventional MMA is not absorbed, FMMA-VR additionally improves the hallucination-sensitive metrics by large margins.
Related
- Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
- LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
- A Structure-Preserving and Thermodynamically Compatible Cell-Centered Lagrangian Finite Volume Scheme for Continuum Mechanics
- Determinacy and Modality in Positive Characteristic
- Mathematical Analysis of Backdoor-Induced Melanoma Risk in Modality-Heterogeneous Federated Learning
BibTeX
@article{seo2026flashattentioncompatible,
title = {{FlashAttention-Compatible Modality-Mutual Attention with Virtual Reordering}},
author = {Jeong Woo Seo and Ho-Young Jung},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14213852},
url = {https://www.mdpi.com/2227-7390/14/21/3852},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - FlashAttention-Compatible Modality-Mutual Attention with Virtual Reordering
AU - Jeong Woo Seo
AU - Ho-Young Jung
PY - 2026
DA - 2026/10/22
JO - Mathematics
DO - 10.3390/math14213852
UR - https://www.mdpi.com/2227-7390/14/21/3852
AB - Multimodal large language models (MLLMs) process an interleaved image–text sequence with a single autoregressive backbone under causal attention. Since the causal mask prevents image tokens from attending to the question that follows them, their representations are computed without access to the query, which has been identified as a source of visual hallucination. Modality-mutual attention (MMA) mitigates this issue by unlocking the image-to-question region of the causal mask, and dual-order training (DOT) mitigates it by additionally training on the reversed text-first order. However, MMA turns the global causal mask into a sample-dependent dense mask. Consequently, attention cannot be executed via FlashAttention, which is the most efficient attention path, and falls back to a slower implementation. Furthermore, the positional information in the unlocked region does not match that of the reversed input order that motivates the unlock, that is, the positional semantics are inconsistent. To address both problems, we propose FlashAttention-compatible MMA with virtual reordering (FMMA-VR), which repairs the positional information of the unlocked region as if the two modalities were reordered, without physically reordering the sequence. The proposed method is implemented as a Triton-based region-aware FlashAttention-style kernel. On identically pretrained models, it accelerates the attention path by up to 3.59× over conventional MMA for long sequences during supervised fine-tuning (SFT). In post-SFT evaluations, it stays on par with conventional MMA across ten benchmark metrics under full fine-tuning. Under low-budget LoRA adaptation, where the positional shift of conventional MMA is not absorbed, FMMA-VR additionally improves the hallucination-sensitive metrics by large margins.
SN - 2227-7390
ER -