Solvable dynamics model of in-context learning
Abstract
Pretrained Transformer architectures display a surprising in-context learning ability without additional training. How the learning process guides the network to achieve this intriguing capability remains unsolved. In this study, we focus on a single-layer, single-head linear attention model, where empirical risk minimization is defined for learning. Given a finite amount of training examples, we determine the exact solution of network parameter dynamics, which is linked to the training error and generalization error, characterizing how memorization and generalization emerge from learning and how these transitions are related to curvatures of the empirical risk landscape. Furthermore, we calculate a data-dependent free energy, from which the training and generalization errors in the long-time limit of learning dynamics can be well predicted. Our work adds insights toward understanding the mechanism of in-context learning, especially from a learning dynamics perspective.
Related
- Temporal complexity hierarchies in solvable quantum many-body dynamics
- A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
- Slow Context, Fast Symptoms: Multiscale Temporal Dynamics and Context-Induced Coupling in Psychological Systems
- An all-transistor physical-learning transformer can be trained to perform in-context learning
- In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
BibTeX
@misc{bai2026solvable,
title = {{Solvable dynamics model of in-context learning}},
author = {Ruoran Bai and Haiping Huang},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.00921},
eprint = {2611.00921},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.00921},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Solvable dynamics model of in-context learning
AU - Ruoran Bai
AU - Haiping Huang
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.00921
UR - https://arxiv.org/abs/2611.00921
AB - Pretrained Transformer architectures display a surprising in-context learning ability without additional training. How the learning process guides the network to achieve this intriguing capability remains unsolved. In this study, we focus on a single-layer, single-head linear attention model, where empirical risk minimization is defined for learning. Given a finite amount of training examples, we determine the exact solution of network parameter dynamics, which is linked to the training error and generalization error, characterizing how memorization and generalization emerge from learning and how these transitions are related to curvatures of the empirical risk landscape. Furthermore, we calculate a data-dependent free energy, from which the training and generalization errors in the long-time limit of learning dynamics can be well predicted. Our work adds insights toward understanding the mechanism of in-context learning, especially from a learning dynamics perspective.
ER -