Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Abstract
A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
Related
- Diffusion Transformers are Provably Optimal In-context Generators
- A priori regularity of the reverse heat flow and dimension-dependent complexity of higher-order diffusion samplers
- Direct Intermediate Initialization for Tilted Diffusion Samplers
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers
BibTeX
@misc{adibi2026transformers,
title = {{Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling}},
author = {Arman Adibi and Alireza Jafari and Mohammad Ghavamzadeh and Hadi Daneshmand},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.08981},
eprint = {2609.08981},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.08981},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
AU - Arman Adibi
AU - Alireza Jafari
AU - Mohammad Ghavamzadeh
AU - Hadi Daneshmand
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.08981
UR - https://arxiv.org/abs/2609.08981
AB - A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
ER -