Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study
Abstract
In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: \textit{is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?} To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: \textit{sufficient-statistic invariance} and the \textit{martingale property}. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.
Related
- Minimax-Optimality of Posterior Sampling for Reinforcement Learning
- Context-Guided Traversal with Deep Reinforcement Learning for Suspicious Subgraph Detection in Cryptocurrency Transaction Graphs
- Proximal Empirical Bayes for Sparse Regression with Posterior Decision Support
- Improving Pseudo-Multiview Learning via Dual-Sampler Bidirectional Feature Flow
- Empirical Bayes in Bayesian learning: understanding a common practice
BibTeX
@misc{chen2026incontext,
title = {{Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study}},
author = {Weiqin Chen and Xinlong Du and Gesualdo Scutari and Harsha Honnappa},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28977},
eprint = {2610.28977},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28977},
}
RIS
TY - UNPB
TI - Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study
AU - Weiqin Chen
AU - Xinlong Du
AU - Gesualdo Scutari
AU - Harsha Honnappa
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28977
UR - https://arxiv.org/abs/2610.28977
AB - In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: \textit{is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?} To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: \textit{sufficient-statistic invariance} and the \textit{martingale property}. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.
ER -