Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

Abstract

In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass. Under bounded-data assumptions, we construct a single-head transformer whose forward pass approximately implements \textit{preconditioned Richardson iteration} on the associated kernel system. The construction uses O(log(1/ε)) blocks and MLP width O(√(N/ε)) to achieve ε-accurate prediction for prompts of length N. Our construction reveals a functional decomposition within the transformer architecture: softmax attention produces a row-normalized Gaussian-kernel operator needed for \emph{cross-token} interactions, while MLP layers act locally to approximate the \emph{intra-token} scalar arithmetic required by the update. Empirically, we train GPT-2-style transformers on Gaussian-process regression tasks and observe that they progressively align with the exact Gaussian KRR estimator across depth in terms of both \emph{prediction error} and \emph{induced weights}, with ablations further supporting this trend. Comparisons with classical KRR solvers also show that deeper layers align with later solver iterates. Together, we empirically demonstrate that the pretrained transformers exhibit progressive refinement toward exact Gaussian KRR across depth, and theoretically establish inexact preconditioned Richardson iteration as a concrete mechanism for approximating this predictor within an explicitly constructed softmax-attention transformer.

Active Flux and Polygons: why a stabilisation and how to implement boundary conditions
Remi Abgrall, Yongle Liu
arXiv26 Oct 2026PreprintCC BY-NC-ND 4.0Full textNumerical analysis & scientific computingarXiv
Richardson dispersion and anomalous dissipation for the random velocity model with controlled sweeping
Charles Fefferman, Svitlana Mayboroda
arXiv5 Nov 2026PreprintCC0Full textAnalysis & PDEs Mathematical physicsarXiv
On Chow and Picard groups of Richardson varieties
David Anderson, Sam Payne
arXiv10 Oct 2026PreprintarXiv licenceFull textAlgebra & number theoryarXiv
A finite cover for coefficient positivity of stretched Littlewood-Richardson polynomials in the seven-row, size-thirty box
Maseeh Ghodsi
arXiv13 Sep 2026PreprintCC BY 4.0Full textCombinatorics & discrete mathematicsarXiv
Hypercube Richardson varieties from magic words
Balázs Elek
arXiv15 Oct 2026PreprintarXiv licenceFull textAlgebra & number theory Combinatorics & discrete mathematicsarXiv

Cite

BibTeX RIS

BibTeX

@misc{yan2026transformers,
  title = {{Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression}},
  author = {Mingsong Yan and Dongyang Li and Charles Kulick and Sui Tang},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2605.08475},
  eprint = {2605.08475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.08475},
}

RIS

TY  - UNPB
TI  - Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
AU  - Mingsong Yan
AU  - Dongyang Li
AU  - Charles Kulick
AU  - Sui Tang
PY  - 2026
DA  - 2026/09/25
PB  - arXiv
DO  - 10.48550/arxiv.2605.08475
UR  - https://arxiv.org/abs/2605.08475
AB  - In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass. Under bounded-data assumptions, we construct a single-head transformer whose forward pass approximately implements \textit{preconditioned Richardson iteration} on the associated kernel system. The construction uses O(log(1/ε)) blocks and MLP width O(√(N/ε)) to achieve ε-accurate prediction for prompts of length N. Our construction reveals a functional decomposition within the transformer architecture: softmax attention produces a row-normalized Gaussian-kernel operator needed for \emph{cross-token} interactions, while MLP layers act locally to approximate the \emph{intra-token} scalar arithmetic required by the update. Empirically, we train GPT-2-style transformers on Gaussian-process regression tasks and observe that they progressively align with the exact Gaussian KRR estimator across depth in terms of both \emph{prediction error} and \emph{induced weights}, with ablations further supporting this trend. Comparisons with classical KRR solvers also show that deeper layers align with later solver iterates. Together, we empirically demonstrate that the pretrained transformers exhibit progressive refinement toward exact Gaussian KRR across depth, and theoretically establish inexact preconditioned Richardson iteration as a concrete mechanism for approximating this predictor within an explicitly constructed softmax-attention transformer.
ER  -

Abstract

Related

Cite