Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
Abstract
Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task. Robust foundation models offer a promising alternative: adversarially pretrain a model once and then transfer its robustness to downstream tasks through lightweight adaptation. However, a fundamental question remains open: can robustness acquired during pretraining transfer to unseen tasks without further adversarial training? In this study, we answer this question affirmatively. A single model adversarially pretrained at scale can achieve optimal robustness on new tasks without additional task-specific training. Specifically, we show that, for a family of Gaussian-mixture classification tasks, a sufficiently deep linear transformer adversarially trained across tasks can asymptotically attain the robust Bayes error on previously unseen tasks through in-context learning from clean demonstrations. By contrast, a standardly trained model cannot. We further analyze convergence under gradient flow, an accuracy--robustness trade-off, and demonstration complexity.
Related
- Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks
- Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
- Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
- An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
- When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?
BibTeX
@misc{kumano2026adversarially,
title = {{Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures}},
author = {Soichiro Kumano},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07754},
eprint = {2610.07754},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07754},
}
RIS
TY - UNPB
TI - Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
AU - Soichiro Kumano
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.07754
UR - https://arxiv.org/abs/2610.07754
AB - Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task. Robust foundation models offer a promising alternative: adversarially pretrain a model once and then transfer its robustness to downstream tasks through lightweight adaptation. However, a fundamental question remains open: can robustness acquired during pretraining transfer to unseen tasks without further adversarial training? In this study, we answer this question affirmatively. A single model adversarially pretrained at scale can achieve optimal robustness on new tasks without additional task-specific training. Specifically, we show that, for a family of Gaussian-mixture classification tasks, a sufficiently deep linear transformer adversarially trained across tasks can asymptotically attain the robust Bayes error on previously unseen tasks through in-context learning from clean demonstrations. By contrast, a standardly trained model cannot. We further analyze convergence under gradient flow, an accuracy--robustness trade-off, and demonstration complexity.
ER -