Fine-tuning MLIP foundation models: strategies for accuracy and transferability
- 1Department of Engineering , University of Cambridge , Cambridge , UK
- 2Scientific Computing Department , Science and Technology Facilities Council-Daresbury Laboratory , Warrington , UK
- 3Center for Materials Physics and Technology , U.S. Naval Research Laboratory , Washington , DC , USA
Abstract
Adapting machine-learned interatomic potential (MLIP) foundation models to specialised tasks through fine-tuning is an increasingly important practice, yet systematic guidance on when and how to fine-tune is currently limited. We evaluate seven fine-tuning strategies -- naive full-parameter updates, two layer-freezing variants, Low-Rank Adaptation (LoRA), multihead replay, pseudolabelled replay, and replay combined with LoRA -- across five chemically diverse benchmarks (aqueous NaCl, ice polymorphs, SN2 reactions, SPICE biomolecules, and lithium electrolytes), three generations of foundation models, and training sets spanning five orders of magnitude. To support this evaluation we implement three capabilities in the MACE codebase: LoRA adapted for equivariant message-passing architectures, including both scalar and equivariant linear layers; pseudolabelled replay, which decouples the replay data source from the original pretraining corpus; and model-aware atomic reference energy (E0) reestimation for fine-tuning workflows. We find that foundation model quality, correct E0 initialisation, and well-chosen hyperparameters are prerequisites whose impact routinely exceeds that of the fine-tuning strategy itself. Once these prerequisites are met, most strategies achieve strong target-task accuracy, consistently surpassing models trained from scratch. The practical distinction depends on deployment scope: naive fine-tuning offers the best convergence for single-system applications, while multihead replay -- with either original or pseudolabelled data -- is the only approach tested that consistently preserves out-of-distribution robustness, maintaining both pretraining-distribution accuracy for broader deployment and many-body short-range repulsion.
Related
- A Gentle Push: Fine-Tuning Foundation Models for Complex Oxide Surface Reconstructions
- Full-data accuracy with fewer labels for training and fine-tuning machine-learning force fields
- Using structure to evaluate the accuracy and transferability of machine learning potentials of biomolecules
- Fine-tuning universal machine learning potentials for transition state search in surface catalysis
- Bias in Universal Machine-Learned Interatomic Potentials and Its Effects on Fine-Tuning
BibTeX
@misc{tompa2026finetuning,
title = {{Fine-tuning MLIP foundation models: strategies for accuracy and transferability}},
author = {Tamas Lajos Tompa and Eszter Varga-Umbrich and Ilyes Batatia and Alin M. Elena and Noam Bernstein and Gábor Cśanyi},
howpublished = {arXiv (Cornell University)},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.12704},
url = {https://arxiv.org/abs/2606.12704},
note = {CC BY},
}
RIS
TY - UNPB
TI - Fine-tuning MLIP foundation models: strategies for accuracy and transferability
AU - Tamas Lajos Tompa
AU - Eszter Varga-Umbrich
AU - Ilyes Batatia
AU - Alin M. Elena
AU - Noam Bernstein
AU - Gábor Cśanyi
PY - 2026
DA - 2026/06/10
PB - arXiv (Cornell University)
DO - 10.48550/arxiv.2606.12704
UR - https://arxiv.org/abs/2606.12704
AB - Adapting machine-learned interatomic potential (MLIP) foundation models to specialised tasks through fine-tuning is an increasingly important practice, yet systematic guidance on when and how to fine-tune is currently limited. We evaluate seven fine-tuning strategies -- naive full-parameter updates, two layer-freezing variants, Low-Rank Adaptation (LoRA), multihead replay, pseudolabelled replay, and replay combined with LoRA -- across five chemically diverse benchmarks (aqueous NaCl, ice polymorphs, S_N2 reactions, SPICE biomolecules, and lithium electrolytes), three generations of foundation models, and training sets spanning five orders of magnitude. To support this evaluation we implement three capabilities in the MACE codebase: LoRA adapted for equivariant message-passing architectures, including both scalar and equivariant linear layers; pseudolabelled replay, which decouples the replay data source from the original pretraining corpus; and model-aware atomic reference energy (E0) reestimation for fine-tuning workflows. We find that foundation model quality, correct E0 initialisation, and well-chosen hyperparameters are prerequisites whose impact routinely exceeds that of the fine-tuning strategy itself. Once these prerequisites are met, most strategies achieve strong target-task accuracy, consistently surpassing models trained from scratch. The practical distinction depends on deployment scope: naive fine-tuning offers the best convergence for single-system applications, while multihead replay -- with either original or pseudolabelled data -- is the only approach tested that consistently preserves out-of-distribution robustness, maintaining both pretraining-distribution accuracy for broader deployment and many-body short-range repulsion.
SN - 2331-8422
ER -