In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
Abstract
In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tasks. A kernel learner first maps inputs through a fixed nonlinear feature map and then applies linear attention, whereas a feature learner applies attention to the original input, followed by a learned nonlinear readout. We derive predictions for their memorization and generalization errors using the replica method, retaining the effects of pretraining size, task-pool diversity, and training and inference context lengths. The resulting predictions closely match numerical experiments across a broad range of regimes. Our analysis yields phase diagrams that characterize when each architecture is advantageous as the amount of pretraining data, task diversity, and context lengths vary. We further identify qualitatively different context-length scalings for the two learners. Together, these results clarify how architectural choices interact with the dataset and govern nonlinear in-context learning.
Related
- Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
- Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
- Comparing the use of supervised machine learning variable selection methods in the context of two-group classification in the psychological and health sciences
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Predictive-Context LOO: Bayesian Model Comparison with Pretrained Predictive Learners
BibTeX
@misc{gu2026incontext,
title = {{In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners}},
author = {Haotian Gu and Yizhou Xu and Lenka Zdeborová},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01712},
eprint = {2610.01712},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01712},
}
RIS
TY - UNPB
TI - In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
AU - Haotian Gu
AU - Yizhou Xu
AU - Lenka Zdeborová
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.01712
UR - https://arxiv.org/abs/2610.01712
AB - In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tasks. A kernel learner first maps inputs through a fixed nonlinear feature map and then applies linear attention, whereas a feature learner applies attention to the original input, followed by a learned nonlinear readout. We derive predictions for their memorization and generalization errors using the replica method, retaining the effects of pretraining size, task-pool diversity, and training and inference context lengths. The resulting predictions closely match numerical experiments across a broad range of regimes. Our analysis yields phase diagrams that characterize when each architecture is advantageous as the amount of pretraining data, task diversity, and context lengths vary. We further identify qualitatively different context-length scalings for the two learners. Together, these results clarify how architectural choices interact with the dataset and govern nonlinear in-context learning.
ER -