CompMat-Bench: Benchmarking AI Agents for Computational Materials Science
Abstract
Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In computational materials research, repeating the same expensive simulations across agents and trials can make evaluation impractical. We introduce CompMat-Bench, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study's scientific goal. We reproduce the research steps in advance and assess agents on preparing inputs and analyzing outputs for expensive simulations, so expensive simulations can be avoided during evaluation. The reproduced inputs and results serve as ground truth for grading agents with fixed rules, without an LLM judge. The benchmark supports four evaluation conditions: single tasks and workflows composed of related tasks, each with full or reduced methodological guidance. With full guidance on single tasks, agents based on three LLMs demonstrate the ability to complete individual materials research steps, with pass rates of 66.0-90.4% across 94 tasks. Both longer workflows and reduced guidance can limit agent performance, but in different ways for different agents: they lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance. Failure analysis attributes most failures to scientific errors rather than to errors in software usage. CompMat-Bench provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure modes.
Related
- Can Coding Agents Reproduce Findings in Computational Materials Science?
- Agentic programs: an emerging form of scientific software in computational materials science
- Science Done on a Machine by a Machine: AI Agents in Computational Chemistry
- SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
- From Automated Simulation to Autonomous Discovery: A Hierarchical Framework for Agentic Computational Materials Science
BibTeX
@misc{zhang2026compmatbench,
title = {{CompMat-Bench: Benchmarking AI Agents for Computational Materials Science}},
author = {Chenmu Zhang and Levi Felix and Jun-Jie Zhang and Xingfu Li and Xuelian Jiang and Tao Jiang and Subhendu Mishra and Xixi Qin and Boris Yakobson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.00636},
eprint = {2610.00636},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.00636},
}
RIS
TY - UNPB
TI - CompMat-Bench: Benchmarking AI Agents for Computational Materials Science
AU - Chenmu Zhang
AU - Levi Felix
AU - Jun-Jie Zhang
AU - Xingfu Li
AU - Xuelian Jiang
AU - Tao Jiang
AU - Subhendu Mishra
AU - Xixi Qin
AU - Boris Yakobson
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.00636
UR - https://arxiv.org/abs/2610.00636
AB - Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In computational materials research, repeating the same expensive simulations across agents and trials can make evaluation impractical. We introduce CompMat-Bench, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study's scientific goal. We reproduce the research steps in advance and assess agents on preparing inputs and analyzing outputs for expensive simulations, so expensive simulations can be avoided during evaluation. The reproduced inputs and results serve as ground truth for grading agents with fixed rules, without an LLM judge. The benchmark supports four evaluation conditions: single tasks and workflows composed of related tasks, each with full or reduced methodological guidance. With full guidance on single tasks, agents based on three LLMs demonstrate the ability to complete individual materials research steps, with pass rates of 66.0-90.4% across 94 tasks. Both longer workflows and reduced guidance can limit agent performance, but in different ways for different agents: they lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance. Failure analysis attributes most failures to scientific errors rather than to errors in software usage. CompMat-Bench provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure modes.
ER -