A Multidimensional Evaluation Framework for Artificial Intelligence-Based Ocean State Forecasts
Abstract
Artificial intelligence (AI) based ocean state forecasts now approach or surpass those produced by operational ocean forecasting systems as measured by conventional pointwise metrics. However, these metrics may favor spatially smoothed forecasts and cannot determine whether forecast variables remain physically coherent. Forecast evaluation should therefore move beyond overall pointwise accuracy toward assessing forecast trustworthiness. We propose a multidimensional framework comprising overall pointwise accuracy, physical consistency, and distributional consistency. Overall pointwise accuracy summarizes forecast and reference agreement over all valid samples; physical consistency evaluates dynamical balances using model outputs alone; and distributional consistency evaluates distribution tails and multiscale spatial structures. Three AI models were deployed as representative testbeds in a unified operational environment. This comprehensive assessment shows that lower overall pointwise errors cannot preclude substantial dynamical imbalances and suppressed fine-scale variance. Conversely, higher pointwise errors may arise from spatial displacement of coherent structures rather than a poor representation of their amplitude. The results demonstrate that the evaluation of AI ocean forecasting models should be based on the multi-dimensional assessment of forecast trustworthiness, rather than solely relying on a single metric. The study provides a preliminary framework for AI model intercomparison, which is also informative for targeted model refinement and operational implementation.
Related
- An Artificial Intelligence Integrated Geospatial Decision-Support Framework for Scenario-Based Climate-Adaptive Land-Use Optimization and Environmental Risk Reduction
- Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
- Global Evaluation of AI and NWP Precipitation Forecasts During Atmospheric River Events
- GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG
- Evaluating Neural Probabilistic Forecasts of Sparse Earthquake Counts: What Survives a Careful Evaluation
BibTeX
@misc{zu2026multidimensional,
title = {{A Multidimensional Evaluation Framework for Artificial Intelligence-Based Ocean State Forecasts}},
author = {Ziqing Zu and Ye Yuan and Tiejun Ling and Zhi Chen and Xiang Li and Zhiqiang Li and Qiang Wang and Jiangjiang Xia and Jiuke Wang and Yongjun Jia and Yingzhe Cui and Shinan Wang and Zhi-Yuan Zhang and Renbo Pang and Xiaojuan Lin and Yu Zhang and Huier Mo and Ang Li},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.04892},
eprint = {2611.04892},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.04892},
}
RIS
TY - UNPB
TI - A Multidimensional Evaluation Framework for Artificial Intelligence-Based Ocean State Forecasts
AU - Ziqing Zu
AU - Ye Yuan
AU - Tiejun Ling
AU - Zhi Chen
AU - Xiang Li
AU - Zhiqiang Li
AU - Qiang Wang
AU - Jiangjiang Xia
AU - Jiuke Wang
AU - Yongjun Jia
AU - Yingzhe Cui
AU - Shinan Wang
AU - Zhi-Yuan Zhang
AU - Renbo Pang
AU - Xiaojuan Lin
AU - Yu Zhang
AU - Huier Mo
AU - Ang Li
PY - 2026
DA - 2026/11/03
PB - arXiv
DO - 10.48550/arxiv.2611.04892
UR - https://arxiv.org/abs/2611.04892
AB - Artificial intelligence (AI) based ocean state forecasts now approach or surpass those produced by operational ocean forecasting systems as measured by conventional pointwise metrics. However, these metrics may favor spatially smoothed forecasts and cannot determine whether forecast variables remain physically coherent. Forecast evaluation should therefore move beyond overall pointwise accuracy toward assessing forecast trustworthiness. We propose a multidimensional framework comprising overall pointwise accuracy, physical consistency, and distributional consistency. Overall pointwise accuracy summarizes forecast and reference agreement over all valid samples; physical consistency evaluates dynamical balances using model outputs alone; and distributional consistency evaluates distribution tails and multiscale spatial structures. Three AI models were deployed as representative testbeds in a unified operational environment. This comprehensive assessment shows that lower overall pointwise errors cannot preclude substantial dynamical imbalances and suppressed fine-scale variance. Conversely, higher pointwise errors may arise from spatial displacement of coherent structures rather than a poor representation of their amplitude. The results demonstrate that the evaluation of AI ocean forecasting models should be based on the multi-dimensional assessment of forecast trustworthiness, rather than solely relying on a single metric. The study provides a preliminary framework for AI model intercomparison, which is also informative for targeted model refinement and operational implementation.
ER -