Distillation of Synthetic Data for Time Series Foundation Models
Abstract
Time series foundation models (TSFMs) are increasingly pre-trained on synthetically generated time series trajectories, where the data generating process is known. Current pre-training recipes are based on loss objectives which compare TSFM outputs to realized future values of each trajectory. We instead propose loss objectives which compare TSFM outputs to the conditional forecast distribution of each trajectory, a procedure we call synthetic data distillation (SDD). SDD corresponds to a Rao-Blackwellization of the training objective, in that it leaves the expectation of stochastic gradients unchanged while provably reducing the covariance of the stochastic gradient under the Loewner partial ordering. We empirically validate SDD on a TSFM model family of sizes from $4$M to $2.5$B parameters, and observe faster convergence of validation loss at every model size: on Gaussian Process data, SDD attains or improves upon the Status Quo loss whilst requiring 10%−40% less training iterations.
Related
- Latent Inference-Time Guidance of Time Series Foundation Models
- Cadence: Error-Bounded Lossy Compression of Demand Time Series with a Time-Series Foundation Model
- SGA: Uncertainty Quantification for Multi-Step Forecasting in Time Series Foundation Models
- RACE: Residual-Aware Test-Time Adaptation for Neighbor-Rich Time-Series Foundation Model Forecasting
- Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models
BibTeX
@misc{biswas2026distillation,
title = {{Distillation of Synthetic Data for Time Series Foundation Models}},
author = {Niloy Biswas and Noureddine El Karoui},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.09586},
eprint = {2609.09586},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09586},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Distillation of Synthetic Data for Time Series Foundation Models
AU - Niloy Biswas
AU - Noureddine El Karoui
PY - 2026
DA - 2026/09/14
PB - arXiv
DO - 10.48550/arxiv.2609.09586
UR - https://arxiv.org/abs/2609.09586
AB - Time series foundation models (TSFMs) are increasingly pre-trained on synthetically generated time series trajectories, where the data generating process is known. Current pre-training recipes are based on loss objectives which compare TSFM outputs to realized future values of each trajectory. We instead propose loss objectives which compare TSFM outputs to the conditional forecast distribution of each trajectory, a procedure we call synthetic data distillation (SDD). SDD corresponds to a Rao-Blackwellization of the training objective, in that it leaves the expectation of stochastic gradients unchanged while provably reducing the covariance of the stochastic gradient under the Loewner partial ordering. We empirically validate SDD on a TSFM model family of sizes from $4$M to $2.5$B parameters, and observe faster convergence of validation loss at every model size: on Gaussian Process data, SDD attains or improves upon the Status Quo loss whilst requiring 10%−40% less training iterations.
ER -