Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
Abstract
Autoregressive models are trained to predict a system's behavior one step at a time, and recursive generation allows the learned dynamics to unfold over long horizons. To what extent can such dynamics learned from local observations recover broader organization of an underlying system that was only partially observed during training? Here we study small autoregressive transformers trained from scratch on trajectories sampled from restricted parameter regimes of several non-linear dynamical systems, including logistic and sine maps, the Lorenz system, and the generalized Hopf system, with control parameters and state trajectories represented as sequences of continuous tokens. Under closed-loop evaluation at parameters far outside the training distribution, the models can recover self-similar period-doubling cascades, chaotic dynamics, and attractor structures with remarkable visual and numerical fidelity. For the logistic map, a transformer reproduces successive period doublings up to period 128, yielding a finite-order scaling ratio of 4.6687, matching the Feigenbaum constant to within 5×10−4. We further investigate how these structures emerge over the course of training, and reveal with causal interventions how control-parameter information is processed through attention into state prediction and shapes the resulting closed-loop dynamics. These results suggest that a surprisingly narrow window into a system's local behavior may suffice for autoregressive transformers to generalize to its unseen global dynamical organization.
Related
- Extrapolation and explainability in hadron mass predictions with machine learning
- A Unified Perspective on the Dynamics of Deep Transformers
- When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers
- GPU implementation of mixed quantum-classical Liouville molecular dynamics without momentum jump
- Ontology without states: PBR as an argument for dynamics-first realism
BibTeX
@misc{liu2026learning,
title = {{Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers}},
author = {Yilun Liu and Yi Zhang and Ganyu Wu and Sikuan Yan and Mengyue Wang and Alois Knoll and Volker Tresp and Yunpu Ma},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38814},
eprint = {2609.38814},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38814},
}
RIS
TY - UNPB
TI - Learning Chaos Without Seeing Chaos: Extrapolation of Global Dynamics in Autoregressive Transformers
AU - Yilun Liu
AU - Yi Zhang
AU - Ganyu Wu
AU - Sikuan Yan
AU - Mengyue Wang
AU - Alois Knoll
AU - Volker Tresp
AU - Yunpu Ma
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.38814
UR - https://arxiv.org/abs/2609.38814
AB - Autoregressive models are trained to predict a system's behavior one step at a time, and recursive generation allows the learned dynamics to unfold over long horizons. To what extent can such dynamics learned from local observations recover broader organization of an underlying system that was only partially observed during training? Here we study small autoregressive transformers trained from scratch on trajectories sampled from restricted parameter regimes of several non-linear dynamical systems, including logistic and sine maps, the Lorenz system, and the generalized Hopf system, with control parameters and state trajectories represented as sequences of continuous tokens. Under closed-loop evaluation at parameters far outside the training distribution, the models can recover self-similar period-doubling cascades, chaotic dynamics, and attractor structures with remarkable visual and numerical fidelity. For the logistic map, a transformer reproduces successive period doublings up to period 128, yielding a finite-order scaling ratio of 4.6687, matching the Feigenbaum constant to within 5×10^−4. We further investigate how these structures emerge over the course of training, and reveal with causal interventions how control-parameter information is processed through attention into state prediction and shapes the resulting closed-loop dynamics. These results suggest that a surprisingly narrow window into a system's local behavior may suffice for autoregressive transformers to generalize to its unseen global dynamical organization.
ER -