Stability of Measure-to-Measure Transformers on Sub-Gaussian Data
Abstract
Transformers have exhibited impressive empirical success across various domains, but their theoretical foundations remain less developed. This work constitutes a mathematical study of the measure-to-measure operators defined by transformers. We show that transformers map sub-Gaussian inputs to sub-Gaussian outputs; this ensures that taking arbitrary-length compositions of the softmax operator is well-defined. We then show that transformers are Hölder continuous with respect to the 1-Wasserstein distance on appropriate spaces of sub-Gaussian inputs. This allows us to establish estimates on the error propagation along a transformer between a sub-Gaussian input and its empirical approximation. We also study a mean-field analog of the cross-attention mechanism, which is an operator from a pair of probability measures to a single probability measure. We show that cross-attention exhibits different Hölder regularity and sample-complexity in its two input arguments. Last, we apply our results to deduce approximation guarantees for measure-to-measure transformers. Together, these results provide a firm stability and finite-sample theory for transformers on sub-Gaussian data.
Related
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
- Convergence Rates for Lasserre's Upper-Bound Hierarchy on ℝd with Gaussian Measure
- Continuous logistic Gaussian random measure fields for spatial distributional modelling
- Depth Separation for Neural Networks under Gaussian Measure
BibTeX
@misc{cole2026stability,
title = {{Stability of Measure-to-Measure Transformers on Sub-Gaussian Data}},
author = {Frank Cole and Nicholas H. Nelsen and Takashi Furuya},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07717},
eprint = {2610.07717},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07717},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Stability of Measure-to-Measure Transformers on Sub-Gaussian Data
AU - Frank Cole
AU - Nicholas H. Nelsen
AU - Takashi Furuya
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.07717
UR - https://arxiv.org/abs/2610.07717
AB - Transformers have exhibited impressive empirical success across various domains, but their theoretical foundations remain less developed. This work constitutes a mathematical study of the measure-to-measure operators defined by transformers. We show that transformers map sub-Gaussian inputs to sub-Gaussian outputs; this ensures that taking arbitrary-length compositions of the softmax operator is well-defined. We then show that transformers are Hölder continuous with respect to the 1-Wasserstein distance on appropriate spaces of sub-Gaussian inputs. This allows us to establish estimates on the error propagation along a transformer between a sub-Gaussian input and its empirical approximation. We also study a mean-field analog of the cross-attention mechanism, which is an operator from a pair of probability measures to a single probability measure. We show that cross-attention exhibits different Hölder regularity and sample-complexity in its two input arguments. Last, we apply our results to deduce approximation guarantees for measure-to-measure transformers. Together, these results provide a firm stability and finite-sample theory for transformers on sub-Gaussian data.
ER -