A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Models
Abstract
Graph neural networks (GNNs) are routinely employed for spatiotemporal forecasting, yet their performance across widely used benchmark datasets is inconsistent. Here, we perform an audit of dataset properties and baseline models to assess the quality of the benchmarks, and the robustness of the conclusions drawn from them. Using classical statistical tools, we characterise spatiotemporal lagged dependencies in benchmarks, and examine how temporal differencing changes these relationships and affects model rankings. Motivated by this, we re-evaluate temporal linear baselines, significantly reducing the apparent gains from GNNs on several benchmarks, and surpassing GNNs on others. Suspecting that GNNs struggle to extract linear, node-wise signals, we find that supplying them with autoregressive residuals improves their performance particularly on non-traffic benchmarks. Finally, controlled synthetic experiments reveal that GNNs are sensitive to heterogeneity in temporal dynamics and spatial graph interactions. Together, our findings demonstrate that baseline specification, data pre-processing and system heterogeneity shape the interpretations drawn from benchmark rankings, informing the design and robust evaluation of GNNs.
Related
- Measuring Structure in Graph Benchmark Datasets Using Graph Invariants
- Below what training size do deep tabular generators stop beating trivial baselines? A preregistered benchmark on a size ladder of clinical and standard datasets
- TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text
- Event-Aware Spatiotemporal Precipitation Forecasting with Geographic Context and Physics-Guided Regularization
- Beyond Point Forecasts: A Survey on Probabilistic Forecasting for Time Series and Spatiotemporal Data
BibTeX
@misc{martin2026critical,
title = {{A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Models}},
author = {Kenneth Martin and Simon Heilig and Asja Fischer and Michel F. C. Haddad and Adam M. Sykulski and Moshe Eliasof},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.20980},
eprint = {2608.20980},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.20980},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Models
AU - Kenneth Martin
AU - Simon Heilig
AU - Asja Fischer
AU - Michel F. C. Haddad
AU - Adam M. Sykulski
AU - Moshe Eliasof
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2608.20980
UR - https://arxiv.org/abs/2608.20980
AB - Graph neural networks (GNNs) are routinely employed for spatiotemporal forecasting, yet their performance across widely used benchmark datasets is inconsistent. Here, we perform an audit of dataset properties and baseline models to assess the quality of the benchmarks, and the robustness of the conclusions drawn from them. Using classical statistical tools, we characterise spatiotemporal lagged dependencies in benchmarks, and examine how temporal differencing changes these relationships and affects model rankings. Motivated by this, we re-evaluate temporal linear baselines, significantly reducing the apparent gains from GNNs on several benchmarks, and surpassing GNNs on others. Suspecting that GNNs struggle to extract linear, node-wise signals, we find that supplying them with autoregressive residuals improves their performance particularly on non-traffic benchmarks. Finally, controlled synthetic experiments reveal that GNNs are sensitive to heterogeneity in temporal dynamics and spatial graph interactions. Together, our findings demonstrate that baseline specification, data pre-processing and system heterogeneity shape the interpretations drawn from benchmark rankings, informing the design and robust evaluation of GNNs.
ER -