Ensemble reliability and the signal‐to‐noise paradox in ECMWF subseasonal forecasts
- 1ECMWF Reading UK
Abstract
Ensemble forecasts sometimes exhibit counterintuitive statistical properties, such that the correlation between ensemble means and observations () exceeds the correlation between ensemble means and individual members (), and thus models seem to underestimate the predictability of the real world. This behaviour has been interpreted as a “signal‐to‐noise paradox” (SNP), which is commonly diagnosed using the ratio of predictable components (). Here, we emphasise the links between ensemble‐size‐invariant estimates of RPC and other metrics of ensemble reliability and derive a general closed‐form expression for RPC in terms of , the spread–error ratio (SER), and the total variance ratio (VR). Physical constraints on the admissible solutions (i.e., real‐valued and non‐negative variances) provide a mechanism to identify statistically paradoxical sample estimates of RPC, , SER, and VR that correspond to combinations that are not possible without sampling uncertainty. We evaluate three large‐scale atmospheric circulation indices in subseasonal reforecasts from the European Centre for Medium‐Range Weather Forecasts (ECMWF) Integrated Forecasting System (IFS). Large‐ensemble North Atlantic Oscillation (NAO) forecasts evaluated over 80 start dates for the period 2001–2020 generally satisfy unbiased reliability criteria within our estimated sampling uncertainties, but still exhibit anomalously high RPC values at some subseasonal lead times. These lead times also coincide with paradoxical combinations of correlation and reliability metrics that are impossible in the large‐sample limit, indicating an important role for sampling uncertainties. Nevertheless, wintertime NAO indices averaged over days 16–45 exhibit more robust evidence for unreliability characterised by , suggesting that SNP‐like behaviour observed in daily data during the period 2001–2020 is not attributable solely to sampling artefacts. However, these results do not generalise to reduced ensemble size configurations of the same IFS model evaluated over 3120 start dates for the period 1959–2023. In these extended reforecasts, daily NAO indices are well calibrated and for all subseasonal lead times.
Related
- Stress-Testing Dynamical and Generative Downscaling Using Subseasonal Extreme Precipitation Forecasts
- Multivariate phase-space interpretation of ensemble forecasts: A comparison of physical and AI models
- Enhancing Tropical Cyclone Prediction Through AI Weather Model Ensemble Forecasts and Multi‐Source Data Integration
- Precursors, Pathways, and State-Dependent Reliability of Long-range ENSO Prediction in the CESM2 Seasonal-to-Multiyear Large Ensemble
- Estimating ensemble-mean anomaly correlations and the ratio of predictable components using an unbiased estimate of the predictable signal variance
BibTeX
@article{roberts2026ensemble,
title = {{Ensemble reliability and the signal‐to‐noise paradox in ECMWF subseasonal forecasts}},
author = {Christopher David Roberts and Frédéric Vitart},
journal = {Quarterly Journal of the Royal Meteorological Society},
year = {2026},
month = may,
doi = {10.1002/qj.70224},
url = {https://doi.org/10.1002/qj.70224},
note = {CC BY},
}
RIS
TY - JOUR
TI - Ensemble reliability and the signal‐to‐noise paradox in ECMWF subseasonal forecasts
AU - Christopher David Roberts
AU - Frédéric Vitart
PY - 2026
DA - 2026/05/26
JO - Quarterly Journal of the Royal Meteorological Society
DO - 10.1002/qj.70224
UR - https://doi.org/10.1002/qj.70224
AB - Ensemble forecasts sometimes exhibit counterintuitive statistical properties, such that the correlation between ensemble means and observations () exceeds the correlation between ensemble means and individual members (), and thus models seem to underestimate the predictability of the real world. This behaviour has been interpreted as a “signal‐to‐noise paradox” (SNP), which is commonly diagnosed using the ratio of predictable components (). Here, we emphasise the links between ensemble‐size‐invariant estimates of RPC and other metrics of ensemble reliability and derive a general closed‐form expression for RPC in terms of , the spread–error ratio (SER), and the total variance ratio (VR). Physical constraints on the admissible solutions (i.e., real‐valued and non‐negative variances) provide a mechanism to identify statistically paradoxical sample estimates of RPC, , SER, and VR that correspond to combinations that are not possible without sampling uncertainty. We evaluate three large‐scale atmospheric circulation indices in subseasonal reforecasts from the European Centre for Medium‐Range Weather Forecasts (ECMWF) Integrated Forecasting System (IFS). Large‐ensemble North Atlantic Oscillation (NAO) forecasts evaluated over 80 start dates for the period 2001–2020 generally satisfy unbiased reliability criteria within our estimated sampling uncertainties, but still exhibit anomalously high RPC values at some subseasonal lead times. These lead times also coincide with paradoxical combinations of correlation and reliability metrics that are impossible in the large‐sample limit, indicating an important role for sampling uncertainties. Nevertheless, wintertime NAO indices averaged over days 16–45 exhibit more robust evidence for unreliability characterised by , suggesting that SNP‐like behaviour observed in daily data during the period 2001–2020 is not attributable solely to sampling artefacts. However, these results do not generalise to reduced ensemble size configurations of the same IFS model evaluated over 3120 start dates for the period 1959–2023. In these extended reforecasts, daily NAO indices are well calibrated and for all subseasonal lead times.
SN - 0035-9009
SN - 1477-870X
ER -