Strategic Governance of AI Models in Earth Science
Abstract
AI foundation models pretrained on weather and climate data are increasingly fine-tuned to Earth science tasks well beyond weather forecasting. Their development and adoption are outpacing the scientific community's ability to evaluate them. These models are judged almost entirely by benchmark skill metrics, which measure how closely a forecast reproduces a reference product but not whether a model represents the physical processes governing the system it predicts. Forecast skill and physical reliability are therefore distinct properties. The distinction is most consequential under the nonstationary conditions of a changing climate for which these models were never trained. We identify five priorities for the physical evaluation of AI models in Earth science from task-specific emulators to foundation models, spanning training data, fine-tuning, behavioral testing, mechanistic interpretability, and output validation. We recommend three activities for the coming decade: 1) open AI-ready evaluation datasets, 2) a shared reporting standard for physics-based evaluation, and 3) a dedicated research program on the safety of these models.
Related
- From mercury to plastics: Earth system science and governance strategies
- Strategic science is needed to mitigate warming-induced emissions and bioclimate feedbacks effectively and efficiently
- How to Use Score-Based Diffusion in Earth System Science: A Satellite Nowcasting Example
- Shock Detectives: classifying MMS observations of Earth’s magnetosheath via citizen science
- Energy-Aware Earth System Science: A Community Roadmap
BibTeX
@misc{kelp2026strategic,
title = {{Strategic Governance of AI Models in Earth Science}},
author = {Makoto Kelp and Amirhossein Arzani and Patricia Castellanos and Paul Griffiths and Ivan Higuera-Mendieta and Manuel Perez-Carrasco and Viral Shah and Patrick Obin Sturm and James Weber},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10560},
eprint = {2610.10560},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10560},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Strategic Governance of AI Models in Earth Science
AU - Makoto Kelp
AU - Amirhossein Arzani
AU - Patricia Castellanos
AU - Paul Griffiths
AU - Ivan Higuera-Mendieta
AU - Manuel Perez-Carrasco
AU - Viral Shah
AU - Patrick Obin Sturm
AU - James Weber
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.10560
UR - https://arxiv.org/abs/2610.10560
AB - AI foundation models pretrained on weather and climate data are increasingly fine-tuned to Earth science tasks well beyond weather forecasting. Their development and adoption are outpacing the scientific community's ability to evaluate them. These models are judged almost entirely by benchmark skill metrics, which measure how closely a forecast reproduces a reference product but not whether a model represents the physical processes governing the system it predicts. Forecast skill and physical reliability are therefore distinct properties. The distinction is most consequential under the nonstationary conditions of a changing climate for which these models were never trained. We identify five priorities for the physical evaluation of AI models in Earth science from task-specific emulators to foundation models, spanning training data, fine-tuning, behavioral testing, mechanistic interpretability, and output validation. We recommend three activities for the coming decade: 1) open AI-ready evaluation datasets, 2) a shared reporting standard for physics-based evaluation, and 3) a dedicated research program on the safety of these models.
ER -