Generative sequence modeling for infinite memory processes via predictive states
Abstract
We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimation approach based on the predictive states of a process, possibly with infinite-range memory. We show that our estimator achieves fast convergence rates when the past history can be compressed into a low-dimensional statistic that is sufficient for predicting the future. Specifically, we show that the statistical complexity of the estimation problem is determined by the intrinsic dimension of the predictive state space. We establish guarantees for an instantiation of our method based on deep neural network estimators, and we support these theoretical results with experiments.
Related
- Scalable generative modeling of non-Gaussian spatio-temporal fields via autoregressive Gaussian processes
- Aggregated Posterior Predictive Checks for Generative Modeling
- Infinite-variation Lévy processes are either strongly eroded or abrupt
- Rigidity of symmetric holomorphic functions on infinite-dimensional sequence spaces
- Li--Yorke Chaos Along Any Infinite Sequence: Relative Mixing, Sofic and Rokhlin Entropy versus Naive Entropy
BibTeX
@misc{wiecksosa2026generative,
title = {{Generative sequence modeling for infinite memory processes via predictive states}},
author = {Michael Wieck-Sosa and Cosma Rohilla Shalizi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38524},
eprint = {2609.38524},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38524},
}
RIS
TY - UNPB
TI - Generative sequence modeling for infinite memory processes via predictive states
AU - Michael Wieck-Sosa
AU - Cosma Rohilla Shalizi
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.38524
UR - https://arxiv.org/abs/2609.38524
AB - We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimation approach based on the predictive states of a process, possibly with infinite-range memory. We show that our estimator achieves fast convergence rates when the past history can be compressed into a low-dimensional statistic that is sufficient for predicting the future. Specifically, we show that the statistical complexity of the estimation problem is determined by the intrinsic dimension of the predictive state space. We establish guarantees for an instantiation of our method based on deep neural network estimators, and we support these theoretical results with experiments.
ER -