SciExam for ENSO: Can AI Agents Build Climate Models?
Abstract
Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid. The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way. Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. The simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm-cold asymmetry, an open debate that the task never mentions. Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record and that the information it receives shapes how it builds its model. SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.
Related
- Anatomy of the Indian summer monsoon and ENSO relationship in a state-of-the-art CGCM: role of the tropical Atlantic Ocean
- The relation of ENSO with global surface temperature
- Recent strengthening of eastern Pacific ENSO in the last millennium paleorecord
- Explore the ideology of deep learning in ENSO forecasts
- Greenhouse gas forcing dominates the recent strengthening of North Pacific Meridional mode influence on ENSO
BibTeX
@misc{zhang2026sciexam,
title = {{SciExam for ENSO: Can AI Agents Build Climate Models?}},
author = {Yinling Zhang and Langchen Liu and Dongbin Xiu and Xueyan Zou and Xu Kuang and Mengdi Wang and Shilong Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10513},
eprint = {2610.10513},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10513},
}
RIS
TY - UNPB
TI - SciExam for ENSO: Can AI Agents Build Climate Models?
AU - Yinling Zhang
AU - Langchen Liu
AU - Dongbin Xiu
AU - Xueyan Zou
AU - Xu Kuang
AU - Mengdi Wang
AU - Shilong Liu
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.10513
UR - https://arxiv.org/abs/2610.10513
AB - Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid. The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way. Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. The simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm-cold asymmetry, an open debate that the task never mentions. Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record and that the information it receives shapes how it builds its model. SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.
ER -