Systematic evaluation of LLM-based zero-shot model construction in chemistry and materials science: predictive performance and response reliability
Abstract
Purely data-driven materials exploration is often constrained by limited experimental data, motivating exploration strategies that leverage prior knowledge of the target material system encoded in large language models (LLMs). This paper describes an approach that combines LLM-derived pairwise comparisons with preference learning to construct utility-function models that rank experimental conditions with respect to a target property, without directly using measured target-property values for the system of interest either during LLM inference or model fitting. Across nine chemistry and materials science datasets and six LLMs from two providers, we evaluated predictive performance in terms of pairwise-comparison accuracy and rank agreement, and response reliability in terms of position bias, preference consistency, and abstention behavior. Predictive performance depended more strongly on the dataset than on the choice of LLM, and preference learning generally reproduced target-property rankings more accurately than direct property prediction by LLMs. Position bias and preference consistency varied across dataset–LLM combinations. Among 11 abstention configurations, the ensemble performed best overall across the risk–coverage metrics, with consistency-based and LLM-judge-based configurations also performing relatively well. Our systematic examination has characterized the predictive performance of LLM-based zero-shot modeling and the reliability of the underlying LLM output, providing insights into the potential and challenges of this approach.
Related
- Machine-Learning-Driven Performance Optimization of Nano-Modified Construction Materials: A Bibliometric Analysis and Systematic Review
- Systematic Evaluation of Mixture-Design for Multi-Oxide Magnesium Aluminosilicate Glass Development: Model Comparison, Modifier Interactions, and Predictive Limitations
- Lunar Regolith Simulant–Filled High-Performance Thermoplastic Composites for Lunar Construction and Their Potential for Additive Manufacturing
- Integrated workflow for univariate and multivariate evaluation of batch correction reliability
- Physical, mechanical, and radiation shielding performance of granodiorite for construction applications
BibTeX
@misc{maezawa2026systematic,
title = {{Systematic evaluation of LLM-based zero-shot model construction in chemistry and materials science: predictive performance and response reliability}},
author = {Shunya Maezawa and Tsuyoshi Ueno},
howpublished = {ChemRxiv},
year = {2026},
month = sep,
doi = {10.26434/chemrxiv.15008241/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15008241/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Systematic evaluation of LLM-based zero-shot model construction in chemistry and materials science: predictive performance and response reliability
AU - Shunya Maezawa
AU - Tsuyoshi Ueno
PY - 2026
DA - 2026/09/03
PB - ChemRxiv
DO - 10.26434/chemrxiv.15008241/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15008241/v1
AB - Purely data-driven materials exploration is often constrained by limited experimental data, motivating exploration strategies that leverage prior knowledge of the target material system encoded in large language models (LLMs). This paper describes an approach that combines LLM-derived pairwise comparisons with preference learning to construct utility-function models that rank experimental conditions with respect to a target property, without directly using measured target-property values for the system of interest either during LLM inference or model fitting. Across nine chemistry and materials science datasets and six LLMs from two providers, we evaluated predictive performance in terms of pairwise-comparison accuracy and rank agreement, and response reliability in terms of position bias, preference consistency, and abstention behavior. Predictive performance depended more strongly on the dataset than on the choice of LLM, and preference learning generally reproduced target-property rankings more accurately than direct property prediction by LLMs. Position bias and preference consistency varied across dataset–LLM combinations. Among 11 abstention configurations, the ensemble performed best overall across the risk–coverage metrics, with consistency-based and LLM-judge-based configurations also performing relatively well. Our systematic examination has characterized the predictive performance of LLM-based zero-shot modeling and the reliability of the underlying LLM output, providing insights into the potential and challenges of this approach.
ER -