From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
- 1Peking University , Shenzhen Graduate School
- 2International Digital Economy Academy (IDEA)
Abstract
Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.
Related
- How Chemical Evidence Shapes Mechanistic Reasoning and Prediction by Large Language Models: Fine-Grained Inference in Nucleoside-Derived Supramolecular Gelation
- A Zero-Shot Single-Point Chemical Language Model Mimics Medicinal Chemist Reasoning for Molecular Optimization and STING Inhibitor Development in Alzheimer’s Disease
- Surfactant-Toolkit and SurfBench-200: an open computation toolkit and a benchmark of tool-augmented language-model reasoning in surfactant science
- Mechanism-grounded fragmentation trees enable explicit chemical reasoning for electron ionization mass spectrum prediction
- Human Language Shapes the Emerging Physical Reasoning in Large Language Models
BibTeX
@misc{guo2026from,
title = {{From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models}},
author = {Hongyu Guo and Hao Li (31608) and He Cao and Gongbo Zhang and Li Yuan},
howpublished = {arXiv (Cornell University)},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.03660},
url = {https://arxiv.org/abs/2606.03660},
note = {CC BY-SA},
}
RIS
TY - UNPB
TI - From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
AU - Hongyu Guo
AU - Hao Li (31608)
AU - He Cao
AU - Gongbo Zhang
AU - Li Yuan
PY - 2026
DA - 2026/06/02
PB - arXiv (Cornell University)
DO - 10.48550/arxiv.2606.03660
UR - https://arxiv.org/abs/2606.03660
AB - Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.
SN - 2331-8422
ER -