When Language Models Misinterpret Uncertainty in Earth System Mechanism Attribution
Abstract
Mechanistic attribution in Earth system science requires quantifying uncertainty and communicating its implications. Here, we develop an uncertainty-aware mechanistic attribution pipeline that constructs probabilistic fingerprints from ensemble estimates, including covariance-aware separation of competing mechanisms, and an evidence ladder that systematically varies the information provided to language models (LLMs). We evaluate this framework in an idealized stochastic system and a coupled El Niño model using controlled interventions and ensemble data assimilation. In the idealized system, numerical separation increases GPT-4o's intervention-label agreement from 0.80 to 0.98, but not GPT-4o-mini's. In the coupled system, both LLMs identify seven of eight fingerprint-ambiguous events when given full fingerprints with verbal interpretations; GPT-4o's false ambiguity reports decrease from 10/82 to 1/82, without consistent gains in intervention-label agreement. For eight historical ENSO events, the uncertainty-aware condition agrees with the model-inferred mechanism labels in seven cases. These findings distinguish uncertainty assessment from its communication.
Related
- Attribution-State Conformal Forecasting: Residual-Independent Explanation States for Uncertainty Calibration Under Distribution Shift
- Post-hoc Epistemic Uncertainty for Vision-Language-Action Policies
- llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R
- Credal Large Language Models for Semantic Commitment under Uncertainty
- System Level Synthesis for Fast Chance-Constrained Post-Fault Ascent Replanning under Persistent Propulsion Uncertainty
BibTeX
@misc{behnoudfar2026when,
title = {{When Language Models Misinterpret Uncertainty in Earth System Mechanism Attribution}},
author = {Pouria Behnoudfar and Nan Chen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30501},
eprint = {2610.30501},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30501},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - When Language Models Misinterpret Uncertainty in Earth System Mechanism Attribution
AU - Pouria Behnoudfar
AU - Nan Chen
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.30501
UR - https://arxiv.org/abs/2610.30501
AB - Mechanistic attribution in Earth system science requires quantifying uncertainty and communicating its implications. Here, we develop an uncertainty-aware mechanistic attribution pipeline that constructs probabilistic fingerprints from ensemble estimates, including covariance-aware separation of competing mechanisms, and an evidence ladder that systematically varies the information provided to language models (LLMs). We evaluate this framework in an idealized stochastic system and a coupled El Niño model using controlled interventions and ensemble data assimilation. In the idealized system, numerical separation increases GPT-4o's intervention-label agreement from 0.80 to 0.98, but not GPT-4o-mini's. In the coupled system, both LLMs identify seven of eight fingerprint-ambiguous events when given full fingerprints with verbal interpretations; GPT-4o's false ambiguity reports decrease from 10/82 to 1/82, without consistent gains in intervention-label agreement. For eight historical ENSO events, the uncertainty-aware condition agrees with the model-inferred mechanism labels in seven cases. These findings distinguish uncertainty assessment from its communication.
ER -