Fine‐Tuning a Weather Foundation Model With Lightweight Decoders for Unseen Physical Processes
6 affiliations
- 1ETH AI Center ETH Zurich Zurich Switzerland
- 2Seminar for Applied Mathematics ETH Zurich Zurich Switzerland
- 3Swiss Data Science Center ETH Zurich Zurich Switzerland
- 4Institute of Geodesy and Photogrammetry ETH Zurich Zurich Switzerland
- 5Scalable Parallel Computing Lab ETH Zurich Zurich Switzerland
- 6Department of Applied Mathematics and Theoretical Physics University of Cambridge Cambridge UK
Abstract
Recent advances in AI weather forecasting have led to the emergence of so‐called “foundation models”, typically defined by expensive pretraining and minimal fine‐tuning for downstream tasks. However, in the natural sciences, a desirable foundation model should also encode meaningful statistical relationships between the underlying physical variables. This study evaluates the performance of the state‐of‐the‐art Aurora foundation model in predicting hydrological variables, which were not included during pretraining. We introduce a lightweight approach using shallow decoders trained on the latent representations of the pretrained model to predict these new variables. As a baseline, we compare this to fine‐tuning the full model, which allows further optimization of the latent space while incorporating new variables into both inputs and outputs. The decoder‐based approach requires 43% less training time and 53% less memory, while achieving strong accuracy across various hydrological variables and preserving desirable properties of the foundation model, such as autoregressive stability. Notably, decoder accuracy depends on the physical correlation between the new variables and those used during pretraining, indicating that Aurora's latent space captures meaningful physical relationships. The decoders outperform UNet, climatology, and persistence baselines on variables related to pretraining, showing the benefits of the latent space. In this sense, we argue that an important quality metric for foundation models in Earth sciences is their ability to be extended to new variables without a full fine‐tuning. This provides a new perspective for making foundation models more accessible to communities with limited computational resources, while supporting broader adoption in Earth sciences.
Related
- Weather Jiu-Jitsu: Exploring the Feasibility of Control Paradigms in Weather Foundation Models
- Xaurora: Generative Weather Forecasting with Denoising Stochastic Interpolants from a Foundation Model Prior
- PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models
- FastNet: Improving the Physical Consistency of Machine Learning Weather Prediction Models through Loss Function Design
- Python-Fortran Hybrid Programming to Fuse AI and Physical Models: Examples of AI-LDA in climate and weather models (Hf2pMDA_v1.0)
BibTeX
@article{lehmann2026finetuning,
title = {{Fine‐Tuning a Weather Foundation Model With Lightweight Decoders for Unseen Physical Processes}},
author = {Fanny Lehmann and Fırat Özdemir and Benedikt S. Soja and Torsten Hoefler and Siddhartha Mishra and Sebastian Schemm},
journal = {Journal of Geophysical Research Machine Learning and Computation},
year = {2026},
month = jul,
doi = {10.1029/2025jh000837},
url = {https://doi.org/10.1029/2025jh000837},
note = {CC BY},
}
RIS
TY - JOUR
TI - Fine‐Tuning a Weather Foundation Model With Lightweight Decoders for Unseen Physical Processes
AU - Fanny Lehmann
AU - Fırat Özdemir
AU - Benedikt S. Soja
AU - Torsten Hoefler
AU - Siddhartha Mishra
AU - Sebastian Schemm
PY - 2026
DA - 2026/07/23
JO - Journal of Geophysical Research Machine Learning and Computation
DO - 10.1029/2025jh000837
UR - https://doi.org/10.1029/2025jh000837
AB - Recent advances in AI weather forecasting have led to the emergence of so‐called “foundation models”, typically defined by expensive pretraining and minimal fine‐tuning for downstream tasks. However, in the natural sciences, a desirable foundation model should also encode meaningful statistical relationships between the underlying physical variables. This study evaluates the performance of the state‐of‐the‐art Aurora foundation model in predicting hydrological variables, which were not included during pretraining. We introduce a lightweight approach using shallow decoders trained on the latent representations of the pretrained model to predict these new variables. As a baseline, we compare this to fine‐tuning the full model, which allows further optimization of the latent space while incorporating new variables into both inputs and outputs. The decoder‐based approach requires 43% less training time and 53% less memory, while achieving strong accuracy across various hydrological variables and preserving desirable properties of the foundation model, such as autoregressive stability. Notably, decoder accuracy depends on the physical correlation between the new variables and those used during pretraining, indicating that Aurora's latent space captures meaningful physical relationships. The decoders outperform UNet, climatology, and persistence baselines on variables related to pretraining, showing the benefits of the latent space. In this sense, we argue that an important quality metric for foundation models in Earth sciences is their ability to be extended to new variables without a full fine‐tuning. This provides a new perspective for making foundation models more accessible to communities with limited computational resources, while supporting broader adoption in Earth sciences.
SN - 2993-5210
ER -