Chaos and bifurcations in continuous language-model feedback
Abstract
Feeding a pretrained language model the expected embedding of its own next-token distribution turns it into a discrete-time dynamical system with continuous states. Across Qwen2.5, OLMo-2 and Llama-3 models from 0.5B to 72B parameters, temperature and activation steering move this loop between repeating text and chaos, with positive finite-time Lyapunov exponents. The loop's derivative factorizes into the inverse temperature, the covariance of token embeddings under the next-token distribution (a cross-covariance for untied embeddings) and the network's Jacobian. In one model studied in detail, along recorded orbits, the growth that steering adds comes mainly from the readout, where next tokens compete. The periodic windows show the quantitative signatures of low-dimensional maps, from saddle-node scaling and Feigenbaum's constant to shrimp-shaped domains; passage times predicted in advance from the model's derivatives came within one return-map iterate of the measured passage times in 42 of 48 cases over two prospective stages (24 and 18 of 24), and interval arithmetic proves attracting periodic orbits of the full transformer map. Numerical reproducibility depends on the dynamical regime: changing only the batch layout changed the text of 173 of 192 chaotic float64 loops in the smallest model, compared with only 7 of 192 periodic ones. These results show that bifurcation theory can make quantitative predictions about continuous language-model feedback, with rigorous guarantees for selected periodic orbits of the full transformer.
Related
- Synergistic Effects of Behavioral Feedback and Seasonality Generate Chaos in Cooperative Multi-Pathogen Systems
- From elementary processes to chaos: a discrete-time two-stage contagion model and its continuous limit
- Probability bounds on trajectory-level exceedances of quantum speed limits under continuous-measurement feedback
- Information transfer as a new indicator of approaching local bifurcations: saddle-node, pitchfork and Hopf bifurcations
- Extrapolating learned dynamics across bifurcations based on generalized synchronization
BibTeX
@misc{jones2026chaos,
title = {{Chaos and bifurcations in continuous language-model feedback}},
author = {Timothy D. Jones},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.31692},
eprint = {2610.31692},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.31692},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Chaos and bifurcations in continuous language-model feedback
AU - Timothy D. Jones
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.31692
UR - https://arxiv.org/abs/2610.31692
AB - Feeding a pretrained language model the expected embedding of its own next-token distribution turns it into a discrete-time dynamical system with continuous states. Across Qwen2.5, OLMo-2 and Llama-3 models from 0.5B to 72B parameters, temperature and activation steering move this loop between repeating text and chaos, with positive finite-time Lyapunov exponents. The loop's derivative factorizes into the inverse temperature, the covariance of token embeddings under the next-token distribution (a cross-covariance for untied embeddings) and the network's Jacobian. In one model studied in detail, along recorded orbits, the growth that steering adds comes mainly from the readout, where next tokens compete. The periodic windows show the quantitative signatures of low-dimensional maps, from saddle-node scaling and Feigenbaum's constant to shrimp-shaped domains; passage times predicted in advance from the model's derivatives came within one return-map iterate of the measured passage times in 42 of 48 cases over two prospective stages (24 and 18 of 24), and interval arithmetic proves attracting periodic orbits of the full transformer map. Numerical reproducibility depends on the dynamical regime: changing only the batch layout changed the text of 173 of 192 chaotic float64 loops in the smallest model, compared with only 7 of 192 periodic ones. These results show that bifurcation theory can make quantitative predictions about continuous language-model feedback, with rigorous guarantees for selected periodic orbits of the full transformer.
ER -