Uncertainty Quantification in Forecast Comparisons
Abstract
Skill scores, which measure the relative improvement of a forecasting method over a benchmark via consistent scoring functions and proper scoring rules, are a standard tool in forecast evaluation, yet their sampling uncertainty is rarely rigorously quantified. With modern forecasting applications being increasingly multivariate and involving evaluations across multiple horizons, variables, spatial locations, and forecasting methods, standard tools like the pairwise Diebold-Mariano forecast accuracy test or pointwise confidence intervals fail to account for the multiple comparison problem, leading to inflated Type I error rates and invalid joint inference. To address the lack of a coherent, statistically rigorous framework for quantifying uncertainty across these multi-dimensional evaluation problems, we introduce simultaneous confidence bands for expected scores and skill scores. Our framework provides a versatile tool for joint inference that is applicable to any forecast type from mean and quantile to full distributional forecasts. We develop a bootstrap implementation and show that our bands are valid under multivariate extensions of the classical Diebold-Mariano assumptions. We demonstrate the practical utility of the approach in two case studies by quantifying the benefits of time-varying parameter models for macroeconomic forecasting, and by comparing data-driven and physics-based models in probabilistic weather forecasting.
Related
- Uncertainty analysis of a machine learning-aided ensemble forecast for coastal flood maps
- Reduction of uncertainty in near-term climate forecast by combining observations and decadal predictions
- Sampling Uncertainty is Critical for Setting Forecast-Based Anticipatory Action Triggers
- Uncertainty quantification for renewable energy generation from landfill methane
- Rigorous uncertainty quantification of probabilistic AI weather forecasts with conformal prediction
BibTeX
@misc{pohle2026uncertainty,
title = {{Uncertainty Quantification in Forecast Comparisons}},
author = {Marc-Oliver Pohle and Tanja Zahn and Sebastian Lerch},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.03997},
eprint = {2605.03997},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.03997},
}
RIS
TY - UNPB
TI - Uncertainty Quantification in Forecast Comparisons
AU - Marc-Oliver Pohle
AU - Tanja Zahn
AU - Sebastian Lerch
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2605.03997
UR - https://arxiv.org/abs/2605.03997
AB - Skill scores, which measure the relative improvement of a forecasting method over a benchmark via consistent scoring functions and proper scoring rules, are a standard tool in forecast evaluation, yet their sampling uncertainty is rarely rigorously quantified. With modern forecasting applications being increasingly multivariate and involving evaluations across multiple horizons, variables, spatial locations, and forecasting methods, standard tools like the pairwise Diebold-Mariano forecast accuracy test or pointwise confidence intervals fail to account for the multiple comparison problem, leading to inflated Type I error rates and invalid joint inference. To address the lack of a coherent, statistically rigorous framework for quantifying uncertainty across these multi-dimensional evaluation problems, we introduce simultaneous confidence bands for expected scores and skill scores. Our framework provides a versatile tool for joint inference that is applicable to any forecast type from mean and quantile to full distributional forecasts. We develop a bootstrap implementation and show that our bands are valid under multivariate extensions of the classical Diebold-Mariano assumptions. We demonstrate the practical utility of the approach in two case studies by quantifying the benefits of time-varying parameter models for macroeconomic forecasting, and by comparing data-driven and physics-based models in probabilistic weather forecasting.
ER -