Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
Abstract
Softmax attention, at the heart of Transformers, has demonstrated remarkable capabilities. Yet its underlying mechanisms remain only partially understood. Recent theoretical work studies Gaussian prompts, where the infinite-prompt limit reduces softmax attention to a linear map, but also removes the query-dependent selection that distinguishes it from linear attention. This work studies the infinite-prompt limit of softmax attention on Gaussian mixtures, which retain the tractability of Gaussian data while introducing latent structure, multimodality, and nonlinear dependencies. We show that softmax attention can represent and learn, via gradient-based methods, optimal solutions to a range of statistical tasks, including supervised classification and denoising. Our results highlight two complementary capabilities of softmax attention: it can recover linear tasks as effectively as its simpler linear counterpart, while also exploiting query-dependent context selection to solve nonlinear tasks beyond the reach of linear attention.
Related
- Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
- In-context classification: Calibration and noise in linear attention with softmax readout
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Notes on Uniform Policy Score Bounds and Lipschitz Continuity for Truncated Gaussian, Product Cauchy, Boltzmann, and Softmax Policies
- RoPE attention is an exact forward-pass gradient step with softmax intact
BibTeX
@misc{gabet2026softmax,
title = {{Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must}},
author = {Simon Gabet and Etienne Boursier and Claire Boyer},
howpublished = {HAL},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11798},
eprint = {2610.11798},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05785598v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
AU - Simon Gabet
AU - Etienne Boursier
AU - Claire Boyer
PY - 2026
DA - 2026/10/08
PB - HAL
DO - 10.48550/arxiv.2610.11798
UR - https://hal.science/hal-05785598v1
AB - Softmax attention, at the heart of Transformers, has demonstrated remarkable capabilities. Yet its underlying mechanisms remain only partially understood. Recent theoretical work studies Gaussian prompts, where the infinite-prompt limit reduces softmax attention to a linear map, but also removes the query-dependent selection that distinguishes it from linear attention. This work studies the infinite-prompt limit of softmax attention on Gaussian mixtures, which retain the tractability of Gaussian data while introducing latent structure, multimodality, and nonlinear dependencies. We show that softmax attention can represent and learn, via gradient-based methods, optimal solutions to a range of statistical tasks, including supervised classification and denoising. Our results highlight two complementary capabilities of softmax attention: it can recover linear tasks as effectively as its simpler linear counterpart, while also exploiting query-dependent context selection to solve nonlinear tasks beyond the reach of linear attention.
ER -