In-context classification: Calibration and noise in linear attention with softmax readout
Abstract
We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its direction from Bernoulli-labeled context examples. We derive an asymptotic prediction for the test logit in a high-dimensional regime with context length proportional to token dimension, and total training prompts proportional to dimension squared. Our formula cleanly separates calibration, test-time context noise, and non-isotropic parameter-estimation noise as interpretable effects on in-context performance. Our analysis also identifies the ridgeless separability regime and addresses the effects of context length, pretraining-set size, regularisation, and sampling temperature on in-context classification performance. The theory's predictions closely agree with finite-dimensional simulations and fully trained linear-attention models, while experiments with standard softmax attention show qualitatively related behaviour beyond the regime assumed by our analysis.
Related
- Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
- RoPE attention is an exact forward-pass gradient step with softmax intact
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
- Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
BibTeX
@misc{rysmakhanov2026incontext,
title = {{In-context classification: Calibration and noise in linear attention with softmax readout}},
author = {Arman Rysmakhanov and Mary Letey and Yue M. Lu and Jacob Zavatone-Veth and Cengiz Pehlevan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17061},
eprint = {2610.17061},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17061},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - In-context classification: Calibration and noise in linear attention with softmax readout
AU - Arman Rysmakhanov
AU - Mary Letey
AU - Yue M. Lu
AU - Jacob Zavatone-Veth
AU - Cengiz Pehlevan
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.17061
UR - https://arxiv.org/abs/2610.17061
AB - We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its direction from Bernoulli-labeled context examples. We derive an asymptotic prediction for the test logit in a high-dimensional regime with context length proportional to token dimension, and total training prompts proportional to dimension squared. Our formula cleanly separates calibration, test-time context noise, and non-isotropic parameter-estimation noise as interpretable effects on in-context performance. Our analysis also identifies the ridgeless separability regime and addresses the effects of context length, pretraining-set size, regularisation, and sampling temperature on in-context classification performance. The theory's predictions closely agree with finite-dimensional simulations and fully trained linear-attention models, while experiments with standard softmax attention show qualitatively related behaviour beyond the regime assumed by our analysis.
ER -