Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning

Abstract

Transformers have emerged as a dominant architecture in modern machine learning, with in-context learning (ICL) enabling models to adapt to new tasks by conditioning on prompts, without any parameter updates. To understand the attention mechanism in transformer blocks, a growing line of theoretical work interprets ICL through the lens of classical statistical estimators. Recent works have established a connection between the attention mechanism and kernel methods, while the kernel function is usually assumed to be fixed. In practice, the query and key matrices in the attention mechanism are trained, which allows the attention mechanism to serve as an adaptive kernel method. In this work, we demonstrate that transformers can act as adaptive anisotropic kernel learners. Specifically, we study ICL for low-rank linear and multi-index nonlinear regressions, and conduct numerical simulations to show that the learned attention scores closely resemble the oracle kernels for these function classes. We then develop a general framework to analyze the generalization error of ICL by connecting transformers to the oracle kernel estimator for each function class. Our theory gives an explicit and constructive implementation of the oracle kernel estimators for these function classes using a wide and shallow transformer architecture, illustrating how the prediction is carried out by the interaction between the query token and the context tokens. Our work complements existing studies by providing numerical and theoretical evidence of how the attention mechanism adapts to anisotropic kernels in both linear and nonlinear settings.

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
Zhongjie Shi, Rongjie Lai, Alexander Cloninger +1
arXiv24 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
Haotian Gu, Yizhou Xu, Lenka Zdeborová
arXiv1 Oct 2026PreprintarXiv licenceFull textStatistics Condensed matterarXiv
Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
Soichiro Kumano
arXiv6 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
Mingsong Yan, Dongyang Li, Charles Kulick +1
arXiv25 Sep 2026PreprintarXiv licenceFull textNumerical analysis & scientific computing Optimisation & controlarXiv
Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers
Jaehee Seo, Jisu Kim
arXiv25 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{shen2026transformers,
  title = {{Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning}},
  author = {Zhaiming Shen and Alexander Cloninger and Rongjie Lai and Wenjing Liao},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.24441},
  eprint = {2610.24441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.24441},
}

RIS

TY  - UNPB
TI  - Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
AU  - Zhaiming Shen
AU  - Alexander Cloninger
AU  - Rongjie Lai
AU  - Wenjing Liao
PY  - 2026
DA  - 2026/10/17
PB  - arXiv
DO  - 10.48550/arxiv.2610.24441
UR  - https://arxiv.org/abs/2610.24441
AB  - Transformers have emerged as a dominant architecture in modern machine learning, with in-context learning (ICL) enabling models to adapt to new tasks by conditioning on prompts, without any parameter updates. To understand the attention mechanism in transformer blocks, a growing line of theoretical work interprets ICL through the lens of classical statistical estimators. Recent works have established a connection between the attention mechanism and kernel methods, while the kernel function is usually assumed to be fixed. In practice, the query and key matrices in the attention mechanism are trained, which allows the attention mechanism to serve as an adaptive kernel method. In this work, we demonstrate that transformers can act as adaptive anisotropic kernel learners. Specifically, we study ICL for low-rank linear and multi-index nonlinear regressions, and conduct numerical simulations to show that the learned attention scores closely resemble the oracle kernels for these function classes. We then develop a general framework to analyze the generalization error of ICL by connecting transformers to the oracle kernel estimator for each function class. Our theory gives an explicit and constructive implementation of the oracle kernel estimators for these function classes using a wide and shallow transformer architecture, illustrating how the prediction is carried out by the interaction between the query token and the context tokens. Our work complements existing studies by providing numerical and theoretical evidence of how the attention mechanism adapts to anisotropic kernels in both linear and nonlinear settings.
ER  -

Abstract

Related

Cite