An Asymptotic Theory of Chain-of-Thought in In-Context Learning
Abstract
Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.
Related
- One-Layer Transformers Provably Learn In-Context K-Nearest Neighbor Prediction with Chain-of-Thought
- Learning to Be Monitored: Causal Bounds and Intervention-Aware Post-Training Recipes for Chain-of-Thought Monitorability
- The Surprising Expressivity of a Single Neuron with Autoregressive Chain-of-Thought
- Learning and Control Beyond Linearity: Towards a Non-asymptotic Theory for Bilinear Systems
- How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis
BibTeX
@misc{takanami2026asymptotic,
title = {{An Asymptotic Theory of Chain-of-Thought in In-Context Learning}},
author = {Kaito Takanami and Cengiz Pehlevan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.03217},
eprint = {2606.03217},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.03217},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An Asymptotic Theory of Chain-of-Thought in In-Context Learning
AU - Kaito Takanami
AU - Cengiz Pehlevan
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2606.03217
UR - https://arxiv.org/abs/2606.03217
AB - Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.
ER -