Nonparametric In-Context Learning and Inference via Measure-Indexed Kernel Ridge Regression
Abstract
In-Context learning enables an estimator, trained across multiple tasks, to adapt to a new unseen task solely through its in-context samples, without any parameter refitting. Despite growing interest in the statistical foundations of in-context learning, two gaps still remain. First, existing convergence results do not characterize the distributional behavior of the in-context learning estimators. Second, classical nonparametric methods are not designed to handle the measure-valued inputs that in-context learning naturally produces. We address both gaps by developing a measure-indexed reproducing kernel Hilbert space framework, in which the adaptivity of the estimator is captured through the dependence of a mean functional on the empirical measure of the context. Within this framework, we propose a measure-indexed kernel ridge regression estimator computed from multiple independent tasks and establish its minimax-optimal convergence rate. We further prove that the proposed estimator is pointwise asymptotically normal, enabling the construction of confidence intervals for the mean and prediction intervals for the response on an unseen task. Notably, the estimator achieves valid uncertainty quantification without any refitting when adapting to a new task. As a special case, our framework recovers classical distributed nonparametric regression when all tasks share the same underlying function. The proposed framework is illustrated through a latent Gaussian process example.
Related
- Scalable Nonparametric Change-Plane Regression with Random Features: Estimation and Inference
- Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers
- Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
- Flexible nonparametric inference for causal effects under the front-door model
- Recursive moment updates for scalable Bayesian nonparametric regression
BibTeX
@misc{shang2026nonparametric,
title = {{Nonparametric In-Context Learning and Inference via Measure-Indexed Kernel Ridge Regression}},
author = {Zuofeng Shang and Tianyang Hu and Fan Yang and Yingcong Li},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28056},
eprint = {2610.28056},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28056},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Nonparametric In-Context Learning and Inference via Measure-Indexed Kernel Ridge Regression
AU - Zuofeng Shang
AU - Tianyang Hu
AU - Fan Yang
AU - Yingcong Li
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28056
UR - https://arxiv.org/abs/2610.28056
AB - In-Context learning enables an estimator, trained across multiple tasks, to adapt to a new unseen task solely through its in-context samples, without any parameter refitting. Despite growing interest in the statistical foundations of in-context learning, two gaps still remain. First, existing convergence results do not characterize the distributional behavior of the in-context learning estimators. Second, classical nonparametric methods are not designed to handle the measure-valued inputs that in-context learning naturally produces. We address both gaps by developing a measure-indexed reproducing kernel Hilbert space framework, in which the adaptivity of the estimator is captured through the dependence of a mean functional on the empirical measure of the context. Within this framework, we propose a measure-indexed kernel ridge regression estimator computed from multiple independent tasks and establish its minimax-optimal convergence rate. We further prove that the proposed estimator is pointwise asymptotically normal, enabling the construction of confidence intervals for the mean and prediction intervals for the response on an unseen task. Notably, the estimator achieves valid uncertainty quantification without any refitting when adapting to a new task. As a special case, our framework recovers classical distributed nonparametric regression when all tasks share the same underlying function. The proposed framework is illustrated through a latent Gaussian process example.
ER -