What can linear attention learn from nonlinear teachers in-context?
Abstract
Linear attention is a tractable model for understanding the mechanisms governing in-context learning in transformers. For linear regression tasks, recent asymptotic analyses have characterised its learning and generalisation behaviour. We extend this theory to nonlinear single-index targets, $y=f(x^\top w)+\varepsilon $. Our main result establishes a nonlinearity-noise equivalence: linear attention extracts only the linear Hermite component of f, while the remaining nonlinear structure contributes to the generalisation error as effective noise. This reduction allows results from the corresponding linear theory to be transferred to nonlinear tasks. We illustrate its implications for finite pretraining data and for the transition from task memorisation to task generalisation as task diversity increases. These results identify a limitation of the reduced linear-attention model and provide a tractable starting point for studying nonlinear in-context learning.
Related
- In-context classification: Calibration and noise in linear attention with softmax readout
- One-Layer Transformers Provably Learn In-Context K-Nearest Neighbor Prediction with Chain-of-Thought
- Cognitive Load-Informed Attention Budgets Across Self-Attention and Token–Context Attention
- Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
- Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms
BibTeX
@misc{letey2026what,
title = {{What can linear attention learn from nonlinear teachers in-context?}},
author = {Mary Letey and Arman Rysmakhanov and Yue M. Lu and Cengiz Pehlevan and Jacob Zavatone-Veth},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10761},
eprint = {2610.10761},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10761},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - What can linear attention learn from nonlinear teachers in-context?
AU - Mary Letey
AU - Arman Rysmakhanov
AU - Yue M. Lu
AU - Cengiz Pehlevan
AU - Jacob Zavatone-Veth
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10761
UR - https://arxiv.org/abs/2610.10761
AB - Linear attention is a tractable model for understanding the mechanisms governing in-context learning in transformers. For linear regression tasks, recent asymptotic analyses have characterised its learning and generalisation behaviour. We extend this theory to nonlinear single-index targets, $y=f(x^\top w)+\varepsilon $. Our main result establishes a nonlinearity-noise equivalence: linear attention extracts only the linear Hermite component of f, while the remaining nonlinear structure contributes to the generalisation error as effective noise. This reduction allows results from the corresponding linear theory to be transferred to nonlinear tasks. We illustrate its implications for finite pretraining data and for the transition from task memorisation to task generalisation as task diversity increases. These results identify a limitation of the reduced linear-attention model and provide a tractable starting point for studying nonlinear in-context learning.
ER -