In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
Abstract
Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of \textit{why} one solution might be favoured over another. Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.
Related
- An Asymptotic Theory of Chain-of-Thought in In-Context Learning
- Spectral Theory of an Exactly Solvable Quantum-Optical Oscillator
- Structure theory of finite solvable Lie conformal algebras
- A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
- Allocation Before the Clock: Graph, Set and Game Theory of Professional League of Legends Objective Setups
BibTeX
@misc{cagnetta2026incontext,
title = {{In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model}},
author = {Francesco Cagnetta and Casper L. Christensen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.21901},
eprint = {2610.21901},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.21901},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
AU - Francesco Cagnetta
AU - Casper L. Christensen
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.21901
UR - https://arxiv.org/abs/2610.21901
AB - Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of \textit{why} one solution might be favoured over another. Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.
ER -