Generative Modeling for Cascading Extremes via Autoregressive Transformers
Abstract
Extreme events often occur sequentially, from earthquake-tsunami cascades to successive extreme losses in financial markets. We introduce an extremal token representation that encodes multivariate observations as a categorical sequence. A quiet token records the absence of threshold exceedances; otherwise, the token identifies the exceeding component with the largest marginal probability-scale score. The representation is invariant to componentwise strictly increasing transformations. It preserves the identity and temporal ordering of dominant exceedances but discards their magnitudes and the identities of nondominant simultaneous exceedances. A decoder-only transformer estimates the history-dependent next-token kernel on this representation, and recursive sampling generates future token sequences from the induced joint predictive law. We amortize our generative model by pretraining across simulated process families with varying parameters and dependence persistence. Next-token predictions are evaluated against exact conditional probabilities for held-out categorical processes and particle filter references for heavy-tailed vector autoregressive processes. Applied to ten-second returns for five semiconductor stocks, the pretrained model supports zero-shot transfer, while a few epochs of adaptation improve agreement between observed and generated token frequencies and transition patterns. These results support task-specific tokenization and simulation-based pretraining for modeling which dimensions dominate multivariate extremes.
Related
- Scalable generative modeling of non-Gaussian spatio-temporal fields via autoregressive Gaussian processes
- Market-Informed Networks for Modeling and Forecast Evaluation of Financial Extremes
- Copula-Based Bivariate Kumaraswamy-Teissier Distributions: Modeling Temperature-Rainfall Dependence and Compound Extremes
- A Multi-View Text Classification Framework Combining Transformers and Topic Modeling
- Generative modeling via hierarchical tensor sketching
BibTeX
@misc{carvalho2026generative,
title = {{Generative Modeling for Cascading Extremes via Autoregressive Transformers}},
author = {Miguel de Carvalho and Myung Won Lee and Jordan Richards and Sotirios Sabanis},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.05728},
eprint = {2611.05728},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.05728},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Generative Modeling for Cascading Extremes via Autoregressive Transformers
AU - Miguel de Carvalho
AU - Myung Won Lee
AU - Jordan Richards
AU - Sotirios Sabanis
PY - 2026
DA - 2026/11/03
PB - arXiv
DO - 10.48550/arxiv.2611.05728
UR - https://arxiv.org/abs/2611.05728
AB - Extreme events often occur sequentially, from earthquake-tsunami cascades to successive extreme losses in financial markets. We introduce an extremal token representation that encodes multivariate observations as a categorical sequence. A quiet token records the absence of threshold exceedances; otherwise, the token identifies the exceeding component with the largest marginal probability-scale score. The representation is invariant to componentwise strictly increasing transformations. It preserves the identity and temporal ordering of dominant exceedances but discards their magnitudes and the identities of nondominant simultaneous exceedances. A decoder-only transformer estimates the history-dependent next-token kernel on this representation, and recursive sampling generates future token sequences from the induced joint predictive law. We amortize our generative model by pretraining across simulated process families with varying parameters and dependence persistence. Next-token predictions are evaluated against exact conditional probabilities for held-out categorical processes and particle filter references for heavy-tailed vector autoregressive processes. Applied to ten-second returns for five semiconductor stocks, the pretrained model supports zero-shot transfer, while a few epochs of adaptation improve agreement between observed and generated token frequencies and transition patterns. These results support task-specific tokenization and simulation-based pretraining for modeling which dimensions dominate multivariate extremes.
ER -