Every Fixed Metric Has a Blind Spot: A Learned Atmospheric Critic for Scoring Forecast Realism
Abstract
Despite their high accuracy on point-wise metrics, machine learning weather forecasting models can exhibit different failure modes such as blurring, periodic irregularities, and other unphysical spatial artifacts. This has motivated a variety of metrics to detect known failure cases. Existing metrics fix a representation or transformation in advance, and that choice limits the artifacts they can detect. We propose to train a discriminator for separating reference data from the model's output, and using its output logit to obtain a divergence-like realism score. The discriminator learns whatever separates the model's fields from real weather, adapting to whichever failure mode that model exhibits. We compare our learned atmospheric critic to existing metrics using various synthetic corruptions applied to ERA5 reanalysis data. Our method successfully identifies the corruptions and ranks their severity, while existing metrics fail on at least one corruption. Additionally, we evaluate forecasts from real weather models, and find that the realism score degrades with longer lead times and the metric generally assigns higher realism to numerical models than to machine learning models.
Related
- AIFS Single 1.1.0: an update to ECMWF's machine-learned weather forecast model AIFS
- Explaining patterns learned by neural networks for detection of atmospheric surface fronts in gridded atmospheric data
- EMEPai 1.0.0: A fast regional data-driven air-quality forecast model for Europe
- LFS-SEnDA: a global ocean forecast system with an efficient Sliding-Ensemble-based Data Assimilation scheme
- Uncertainty analysis of a machine learning-aided ensemble forecast for coastal flood maps
BibTeX
@misc{elberkennou2026every,
title = {{Every Fixed Metric Has a Blind Spot: A Learned Atmospheric Critic for Scoring Forecast Realism}},
author = {Younes Elberkennou and Dmitri Demler and Thierry Meier and Luca Rispoli and Fanny Lehmann and Joel Oskarsson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.18381},
eprint = {2609.18381},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.18381},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Every Fixed Metric Has a Blind Spot: A Learned Atmospheric Critic for Scoring Forecast Realism
AU - Younes Elberkennou
AU - Dmitri Demler
AU - Thierry Meier
AU - Luca Rispoli
AU - Fanny Lehmann
AU - Joel Oskarsson
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2609.18381
UR - https://arxiv.org/abs/2609.18381
AB - Despite their high accuracy on point-wise metrics, machine learning weather forecasting models can exhibit different failure modes such as blurring, periodic irregularities, and other unphysical spatial artifacts. This has motivated a variety of metrics to detect known failure cases. Existing metrics fix a representation or transformation in advance, and that choice limits the artifacts they can detect. We propose to train a discriminator for separating reference data from the model's output, and using its output logit to obtain a divergence-like realism score. The discriminator learns whatever separates the model's fields from real weather, adapting to whichever failure mode that model exhibits. We compare our learned atmospheric critic to existing metrics using various synthetic corruptions applied to ERA5 reanalysis data. Our method successfully identifies the corruptions and ranks their severity, while existing metrics fail on at least one corruption. Additionally, we evaluate forecasts from real weather models, and find that the realism score degrades with longer lead times and the metric generally assigns higher realism to numerical models than to machine learning models.
ER -