Landau theory of quenched criticality in linear in-context learning
Abstract
In-context learning (ICL) allows a pretrained model to infer a new task from examples supplied in its prompt without updating its parameters. In linear models of ICL, the prediction error develops a double-descent singularity when the number of pretraining samples becomes comparable to the number of learnable parameters. We formulate this interpolation singularity as a critical phenomenon of a quenched disordered system. By comparing annealed and quenched descriptions of the same linear ICL model, we identify the connected sample-to-sample fluctuations of the learned parameters as the microscopic origin of the singular error. A Landau potential is constructed by integrating the cavity self-consistency equation for the renormalized ridge parameter ξ. The role of (magnetization) order parameter is played by ξ, while the bare ridge parameter λ becomes its conjugate magnetic field. The normalized sample complexity τ acts as a temperature and the double-descent singularity occurs at the critical temperature τc =1. The Landau susceptibility is precisely the quantity that diverges in the fluctuation contribution to the prediction error. The order parameter is closely related to the fraction of zero eigenvalues of the empirical relaxation matrix in the ridgeless limit, which define flat directions in the learning dynamics. The Landau theory is generically cubic in the order parameter with critical exponents (βcr,δcr,γcr)=(1,2,1). In the large-context regime, there appears a pseudogap-like regime characterized by suppressed order parameter. Predictions of the Landau theory are independently confirmed from numerical solutions of the original learning problem with good quantitative agreement. Our results pave the way for solid statistical-physics understanding of the interpolation criticality in linear in-context learning.
Related
- Distinct Signatures of Dynamical and Spectral Criticality in Quenched Quantum Batteries
- An Asymptotic Theory of Chain-of-Thought in In-Context Learning
- Nucleation-type criticality with infinite number of free-energy minima beyond mean-field theory
- Adiabatic Theory Data on Strongly Chirped Dissipative Solitons of the Cubic-Quintic Nonlinear Ginzburg-Landau Equation
- Landau–Khalatnikov theory of anelastic relaxations in spin-crossover materials
BibTeX
@misc{kim2026landau,
title = {{Landau theory of quenched criticality in linear in-context learning}},
author = {Daesik Kim and Sumin Choi and Hyojae Jeon and Jung Hoon Han},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.28059},
eprint = {2608.28059},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.28059},
}
RIS
TY - UNPB
TI - Landau theory of quenched criticality in linear in-context learning
AU - Daesik Kim
AU - Sumin Choi
AU - Hyojae Jeon
AU - Jung Hoon Han
PY - 2026
DA - 2026/08/28
PB - arXiv
DO - 10.48550/arxiv.2608.28059
UR - https://arxiv.org/abs/2608.28059
AB - In-context learning (ICL) allows a pretrained model to infer a new task from examples supplied in its prompt without updating its parameters. In linear models of ICL, the prediction error develops a double-descent singularity when the number of pretraining samples becomes comparable to the number of learnable parameters. We formulate this interpolation singularity as a critical phenomenon of a quenched disordered system. By comparing annealed and quenched descriptions of the same linear ICL model, we identify the connected sample-to-sample fluctuations of the learned parameters as the microscopic origin of the singular error. A Landau potential is constructed by integrating the cavity self-consistency equation for the renormalized ridge parameter ξ. The role of (magnetization) order parameter is played by ξ, while the bare ridge parameter λ becomes its conjugate magnetic field. The normalized sample complexity τ acts as a temperature and the double-descent singularity occurs at the critical temperature τ_c =1. The Landau susceptibility is precisely the quantity that diverges in the fluctuation contribution to the prediction error. The order parameter is closely related to the fraction of zero eigenvalues of the empirical relaxation matrix in the ridgeless limit, which define flat directions in the learning dynamics. The Landau theory is generically cubic in the order parameter with critical exponents (β_cr,δ_cr,γ_cr)=(1,2,1). In the large-context regime, there appears a pseudogap-like regime characterized by suppressed order parameter. Predictions of the Landau theory are independently confirmed from numerical solutions of the original learning problem with good quantitative agreement. Our results pave the way for solid statistical-physics understanding of the interpolation criticality in linear in-context learning.
ER -