INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
Abstract
Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, yet their ability to configure scientific calculations remains poorly understood. Here, we introduce INCARBench, a benchmark for evaluating LLMs on input configuration for the Vienna Ab initio Simulation Package (VASP) through both configuration generation and repair tasks. Evaluating 19 model configurations reveals substantial capability differences among current frontier models. While several models achieve high semantic and policy accuracy, task-critical correctness remains substantially lower, demonstrating that parameter-level correctness does not necessarily imply scientifically valid configurations. Failure analysis shows that errors concentrate in physically coupled settings involving DFT+U, magnetism, and correlated materials, where multiple constraints must be satisfied simultaneously. Repair evaluation further reveals that correcting incorrect settings and preserving already-valid configurations are distinct capabilities, with configuration preservation remaining a major challenge. These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for developing more reliable AI systems for computational materials science.
Related
- Fine-Tuning Small Language Models for Reliable VASP INCAR Generation
- Benchmark Study for Quantum Statevector Simulators for Scientific Computing
- Explore, Then Commit: Measurement-Efficient Scientific Law Discovery with Language Models
- MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection
- From Density to Biopsy Decisions and Malignancy Prediction: A Benchmark Study of Multimodal Large Language Models Against Radiologists in Digital and Contrast-Enhanced Mammography
BibTeX
@misc{shao2026incarbench,
title = {{INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models}},
author = {Bin Shao and Jixiang Li and Xinyue Zhang and Baishun Yang and Zhiyang Liu and Weichao Wang},
howpublished = {arXiv},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.23571},
eprint = {2606.23571},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.23571},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
AU - Bin Shao
AU - Jixiang Li
AU - Xinyue Zhang
AU - Baishun Yang
AU - Zhiyang Liu
AU - Weichao Wang
PY - 2026
DA - 2026/06/23
PB - arXiv
DO - 10.48550/arxiv.2606.23571
UR - https://arxiv.org/abs/2606.23571
AB - Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, yet their ability to configure scientific calculations remains poorly understood. Here, we introduce INCARBench, a benchmark for evaluating LLMs on input configuration for the Vienna Ab initio Simulation Package (VASP) through both configuration generation and repair tasks. Evaluating 19 model configurations reveals substantial capability differences among current frontier models. While several models achieve high semantic and policy accuracy, task-critical correctness remains substantially lower, demonstrating that parameter-level correctness does not necessarily imply scientifically valid configurations. Failure analysis shows that errors concentrate in physically coupled settings involving DFT+U, magnetism, and correlated materials, where multiple constraints must be satisfied simultaneously. Repair evaluation further reveals that correcting incorrect settings and preserving already-valid configurations are distinct capabilities, with configuration preservation remaining a major challenge. These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for developing more reliable AI systems for computational materials science.
ER -