ClimateBench v2.0: Probabilistic Climate Model Benchmarking
Abstract
We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in projecting mid-century regional temperature and precipitation changes. The protocol is designed to evaluate any physics-based, data-driven, or hybrid climate model on equal footing using a common set of observational and out-of-distribution tests. We define three tiers of evaluation. Tier I establishes physical credibility through entry-ticket tests of energy conservation, coupled (co-)variability, and basic forced responses. Tier II scores models against post-2015 observations of surface temperature, precipitation, radiative fluxes, sea ice, and key modes of variability using fair CRPS as the primary probabilistic score, complemented by distributional and ensemble-consistency diagnostics. Tier III tests out-of-distribution generalization through paleoclimate simulations spanning the Last Interglacial, Last Glacial Maximum, and Mid-Holocene, and through perfect-model experiments in which data-driven models must predict the future climate of existing Earth system models from historical data alone. We reserve all observational data after 2015 for testing, and submissions must include multiple ensemble members to enable probabilistic evaluation. This reservation exploits a new opportunity provided by the decade of observations accumulated since the end of the CMIP6 historical experiment, which constitutes an out-of-sample record of forced climate change (and internal variability) for the current generation of models, and we quantify, in an idealized setting, the information it carries about mid-century warming. We provide the evaluation code, observational reference datasets, and perfect-model training data as an open benchmark to drive measurable progress in climate projection across all modeling approaches.
Related
- Regional climate risk assessment from climate models using probabilistic machine learning
- EC-EarthFlow: Probabilistic emulation of daily transient global climate model simulations with flow matching
- PepC-Global: A Basin-Tuned Probabilistic Tropical Cyclone Model with Enhanced Out-of-Sample Skill and Climate-Sensitive Over-Land Decay
- Probabilistic multi-fidelity climate model parameterization for better generalization
- Benchmarking convection-permitting climate simulations for hydrometeorological characterizations: a comparative study of WRF-SAAG and observation-based products in Chile
BibTeX
@misc{watsonparris2026climatebench,
title = {{ClimateBench v2.0: Probabilistic Climate Model Benchmarking}},
author = {Duncan Watson-Parris and Willa Tobin and Aytaç Paçal and Manuel Schlund and V. Balaji and Kevin Bowman and Chris Bretherton and Peter M. Caldwell and Will Chapman and William D. Collins and Gregory S. Elsaesser and Pierre Gentine and Helene Hewitt and Stephan Hoyer and Ralph Keeling and Nikolay Koldunov and David M. Lawrence and Christian Lessig and Daniel J. Lunt and J. David Neelin and Mike Pritchard and Sarah Purkey and Gavin Schmidt and Tapio Schneider and Michael Schulz and Tiffany Shaw and Isla R. Simpson and Graeme Stephens and Aneesh C. Subramanian and Joao Teixeira and Jessica Tierney and Andrew I. L. Williams and Laure Zanna and Veronika Eyring and Rose Yu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.04558},
eprint = {2610.04558},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.04558},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - ClimateBench v2.0: Probabilistic Climate Model Benchmarking
AU - Duncan Watson-Parris
AU - Willa Tobin
AU - Aytaç Paçal
AU - Manuel Schlund
AU - V. Balaji
AU - Kevin Bowman
AU - Chris Bretherton
AU - Peter M. Caldwell
AU - Will Chapman
AU - William D. Collins
AU - Gregory S. Elsaesser
AU - Pierre Gentine
AU - Helene Hewitt
AU - Stephan Hoyer
AU - Ralph Keeling
AU - Nikolay Koldunov
AU - David M. Lawrence
AU - Christian Lessig
AU - Daniel J. Lunt
AU - J. David Neelin
AU - Mike Pritchard
AU - Sarah Purkey
AU - Gavin Schmidt
AU - Tapio Schneider
AU - Michael Schulz
AU - Tiffany Shaw
AU - Isla R. Simpson
AU - Graeme Stephens
AU - Aneesh C. Subramanian
AU - Joao Teixeira
AU - Jessica Tierney
AU - Andrew I. L. Williams
AU - Laure Zanna
AU - Veronika Eyring
AU - Rose Yu
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.04558
UR - https://arxiv.org/abs/2610.04558
AB - We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in projecting mid-century regional temperature and precipitation changes. The protocol is designed to evaluate any physics-based, data-driven, or hybrid climate model on equal footing using a common set of observational and out-of-distribution tests. We define three tiers of evaluation. Tier I establishes physical credibility through entry-ticket tests of energy conservation, coupled (co-)variability, and basic forced responses. Tier II scores models against post-2015 observations of surface temperature, precipitation, radiative fluxes, sea ice, and key modes of variability using fair CRPS as the primary probabilistic score, complemented by distributional and ensemble-consistency diagnostics. Tier III tests out-of-distribution generalization through paleoclimate simulations spanning the Last Interglacial, Last Glacial Maximum, and Mid-Holocene, and through perfect-model experiments in which data-driven models must predict the future climate of existing Earth system models from historical data alone. We reserve all observational data after 2015 for testing, and submissions must include multiple ensemble members to enable probabilistic evaluation. This reservation exploits a new opportunity provided by the decade of observations accumulated since the end of the CMIP6 historical experiment, which constitutes an out-of-sample record of forced climate change (and internal variability) for the current generation of models, and we quantify, in an idealized setting, the information it carries about mid-century warming. We provide the evaluation code, observational reference datasets, and perfect-model training data as an open benchmark to drive measurable progress in climate projection across all modeling approaches.
ER -