Forecast Skill Is Not Decision Skill: Evidence From Weather‐Dependent Decision Tasks
- 1Tübingen AI Center University of Tübingen Tübingen Germany
- 2University of Augsburg Augsburg Germany
Abstract
Standard weather forecast evaluations focus on the forecaster's perspective and on a statistical assessment comparing forecasts and observations. In practice, however, forecasts are used to make decisions, so it seems natural to take the decision‐maker's perspective and quantify the value of a forecast by its ability to improve decision‐making. Decision calibration provides a novel framework for evaluating probabilistic forecast performance at the decision level rather than the forecast level. We evaluate decision calibration to compare a machine learning and a classical numerical weather prediction model on various weather‐dependent decision tasks, though the framework is applicable to any set of forecast models. We find that model performance at the forecast level does not reliably translate to performance in downstream decision‐making: some performance differences only become apparent at the decision level, and even among seemingly similar decision tasks, model rankings can change. Our results confirm that typical forecast evaluations are insufficient for selecting the optimal forecast model for a specific decision task.
Related
- A multicriteria decision analysis framework for generating baseline virtual weather stations in Zimbabwe
- AIFS Single 1.1.0: an update to ECMWF's machine-learned weather forecast model AIFS
- Forecast lead time and the timing of crew deployment for weather-driven power outages using AI weather models
- Aligning evidence and action: The Research-to-Decision (RED) Framework to address modern environmental and ecological challenges
- Can AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones
BibTeX
@article{raeth2026forecast,
title = {{Forecast Skill Is Not Decision Skill: Evidence From Weather‐Dependent Decision Tasks}},
author = {Kornelius Raeth and Nicole Ludwig},
journal = {Meteorological Applications},
year = {2026},
month = sep,
doi = {10.1002/met.70260},
url = {https://doi.org/10.1002/met.70260},
note = {CC BY},
}
RIS
TY - JOUR
TI - Forecast Skill Is Not Decision Skill: Evidence From Weather‐Dependent Decision Tasks
AU - Kornelius Raeth
AU - Nicole Ludwig
PY - 2026
DA - 2026/09/01
JO - Meteorological Applications
DO - 10.1002/met.70260
UR - https://doi.org/10.1002/met.70260
AB - Standard weather forecast evaluations focus on the forecaster's perspective and on a statistical assessment comparing forecasts and observations. In practice, however, forecasts are used to make decisions, so it seems natural to take the decision‐maker's perspective and quantify the value of a forecast by its ability to improve decision‐making. Decision calibration provides a novel framework for evaluating probabilistic forecast performance at the decision level rather than the forecast level. We evaluate decision calibration to compare a machine learning and a classical numerical weather prediction model on various weather‐dependent decision tasks, though the framework is applicable to any set of forecast models. We find that model performance at the forecast level does not reliably translate to performance in downstream decision‐making: some performance differences only become apparent at the decision level, and even among seemingly similar decision tasks, model rankings can change. Our results confirm that typical forecast evaluations are insufficient for selecting the optimal forecast model for a specific decision task.
SN - 1469-8080
SN - 1350-4827
ER -