WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting
Abstract
Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors. This enables scenario analysis: forecasting how a target responds to events that set covariates to chosen future values. Whether they can do this reliably is an open question. We introduce WhatIf-Bench, a benchmark measuring how well forecasters use covariates to forecast a target's response to future events. WhatIf-Bench is semi-synthetic: time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled covariate-target relationships grounded in each system. Each task presents a history of events intervening on covariates, then asks to forecast a future event that either replays seen values (Replay) or sets them to unseen ones (Novel). Varying the number of covariates, historical events, and event length yields 37,665 tasks. Evaluating TSFMs, frontier LLMs, and simple regression baselines, we find the LLMs reach the highest skill (improvement over a naive baseline), ahead of the best TSFMs. Every model forecasts well on Replay futures (48 to 85% skill) but drops to -7 to 50% on Novel futures, with the regression baselines falling the most. Skill on Novel futures also falls as events set more covariates jointly. More historical events raise skill on Novel futures across the board, but large differences remain. WhatIf-Bench reveals generalization to Novel futures as the core challenge, and offers a testbed toward arbitrary scenario analysis.
Related
- When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting
- Hierarchical Time Series Forecasting with Operator-Valued Kernels
- Informed forecasting: Leveraging auxiliary knowledge to boost large language model performance on time series forecasting
- Series-wise Cross-validation for Global Time Series Forecasting
- Coherent Hierarchical Forecasting for Proportion and Discrete Time Series
BibTeX
@misc{ashok2026whatifbench,
title = {{WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting}},
author = {Arjun Ashok and Mihir Parmar and Palash Goyal and Bhavana Dalvi Mishra and Chun-Liang Li and Irina Rish and Alexandre Drouin and Jinsung Yoon and Tomas Pfister},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.26582},
eprint = {2610.26582},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.26582},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting
AU - Arjun Ashok
AU - Mihir Parmar
AU - Palash Goyal
AU - Bhavana Dalvi Mishra
AU - Chun-Liang Li
AU - Irina Rish
AU - Alexandre Drouin
AU - Jinsung Yoon
AU - Tomas Pfister
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.26582
UR - https://arxiv.org/abs/2610.26582
AB - Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors. This enables scenario analysis: forecasting how a target responds to events that set covariates to chosen future values. Whether they can do this reliably is an open question. We introduce WhatIf-Bench, a benchmark measuring how well forecasters use covariates to forecast a target's response to future events. WhatIf-Bench is semi-synthetic: time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled covariate-target relationships grounded in each system. Each task presents a history of events intervening on covariates, then asks to forecast a future event that either replays seen values (Replay) or sets them to unseen ones (Novel). Varying the number of covariates, historical events, and event length yields 37,665 tasks. Evaluating TSFMs, frontier LLMs, and simple regression baselines, we find the LLMs reach the highest skill (improvement over a naive baseline), ahead of the best TSFMs. Every model forecasts well on Replay futures (48 to 85% skill) but drops to -7 to 50% on Novel futures, with the regression baselines falling the most. Skill on Novel futures also falls as events set more covariates jointly. More historical events raise skill on Novel futures across the board, but large differences remain. WhatIf-Bench reveals generalization to Novel futures as the core challenge, and offers a testbed toward arbitrary scenario analysis.
ER -