MatUQ: a benchmark for uncertainty-aware out-of-distribution materials property prediction with graph neural networks
- 1School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, Guangdong, China
- 2School of Data Science and Society, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Abstract
Reliable uncertainty quantification (UQ) for graph neural networks (GNNs) under out-of-distribution (OOD) shifts remains insufficiently characterized in materials discovery. Existing benchmarks based on random splits can overestimate model reliability by underrepresenting structural extrapolation challenges. Here we introduce MatUQ, a benchmark built on structure-aware Smooth Overlap of Atomic Positions Leave-One-Cluster-Out (SOAP-LOCO) splitting, together with a training protocol that combines Deep Evidential Regression (DER) with dropout regularization, for evaluating GNN reliability under structural distribution shifts. Through systematic experiments spanning six materials datasets, twelve GNN architectures, and eight UQ strategies, we find that predictive accuracy and uncertainty quality are distinct capabilities that tend to decouple under OOD evaluation, and that uncertainty-metric leadership is largely non-transferable across datasets and target properties. We further find that as training data become scarce, the optimal strategy shifts from evidential-containing hybrids toward pure ensemble variance across all evaluated architectures, with the architecture holding the distributional optimum shifting correspondingly. Standalone evidential regression rarely attains per-model optima and benefits from hybrid pairing only under specific combinations of data density, inductive bias, and target metric. Monte Carlo dropout is less effective as a standalone uncertainty estimator under the tested configurations but can contribute within hybrid schemes. Overall, MatUQ provides a more rigorous benchmark for assessing uncertainty-aware GNNs in OOD materials discovery.
Related
- Materials Property Prediction with a Site-Resolved Three-Dimensional Voxel Convolution Neural Network
- PolyBench: A Reproducible Open Benchmark Platform for Repeat-Unit-Based Polymer Property Prediction
- When Do Models Win? A Learning Curve Benchmark for Molecular Property Prediction in Low-Data Regimes
- When Do Quantum-Chemical Descriptors Improve Aqueous Solubility Prediction? A Residual-Rescue Benchmark under Chemical Distribution Shift
- Rescuable Stagnation: A Diagnostic for Benchmark Suitability in LLM-Guided Materials Optimization
BibTeX
@article{tan2026matuq,
title = {{MatUQ: a benchmark for uncertainty-aware out-of-distribution materials property prediction with graph neural networks}},
author = {Liqin Tan and Xiean Wang and Yuexin Zou and Pin Chen and Qingsong Zou},
journal = {npj Computational Materials},
year = {2026},
month = aug,
doi = {10.1038/s41524-026-02272-x},
url = {https://doi.org/10.1038/s41524-026-02272-x},
note = {CC BY},
}
RIS
TY - JOUR
TI - MatUQ: a benchmark for uncertainty-aware out-of-distribution materials property prediction with graph neural networks
AU - Liqin Tan
AU - Xiean Wang
AU - Yuexin Zou
AU - Pin Chen
AU - Qingsong Zou
PY - 2026
DA - 2026/08/15
JO - npj Computational Materials
DO - 10.1038/s41524-026-02272-x
UR - https://doi.org/10.1038/s41524-026-02272-x
AB - Reliable uncertainty quantification (UQ) for graph neural networks (GNNs) under out-of-distribution (OOD) shifts remains insufficiently characterized in materials discovery. Existing benchmarks based on random splits can overestimate model reliability by underrepresenting structural extrapolation challenges. Here we introduce MatUQ, a benchmark built on structure-aware Smooth Overlap of Atomic Positions Leave-One-Cluster-Out (SOAP-LOCO) splitting, together with a training protocol that combines Deep Evidential Regression (DER) with dropout regularization, for evaluating GNN reliability under structural distribution shifts. Through systematic experiments spanning six materials datasets, twelve GNN architectures, and eight UQ strategies, we find that predictive accuracy and uncertainty quality are distinct capabilities that tend to decouple under OOD evaluation, and that uncertainty-metric leadership is largely non-transferable across datasets and target properties. We further find that as training data become scarce, the optimal strategy shifts from evidential-containing hybrids toward pure ensemble variance across all evaluated architectures, with the architecture holding the distributional optimum shifting correspondingly. Standalone evidential regression rarely attains per-model optima and benefits from hybrid pairing only under specific combinations of data density, inductive bias, and target metric. Monte Carlo dropout is less effective as a standalone uncertainty estimator under the tested configurations but can contribute within hybrid schemes. Overall, MatUQ provides a more rigorous benchmark for assessing uncertainty-aware GNNs in OOD materials discovery.
SN - 2057-3960
ER -