TSBench: A physics-grounded benchmark for evaluating LLM understanding of chemical reaction mechanisms
Abstract
Understanding a chemical reaction requires mapping a symbolic reactant-product description onto the three-dimensional pathway through which atoms rearrange, yet chemistry benchmarks for large language models (LLMs) largely probe factual knowledge and text-based reasoning. Here we introduce TSBench, a benchmark in which an LLM agent uses structure-editing tools to construct three-dimensional transition-state (TS) guesses verified by an automated quantum-chemical pipeline, yielding a physics-grounded pass/fail verdict. Across 546 evaluations of seven frontier LLMs on 78 elementary reactions, the aggregate success rate rose from 50.4% to 66.8% under diagnosis-driven revision; the best models approached 90% on the simplest reactions, yet performance dropped sharply with mechanistic complexity. Most failed attempts produced locally plausible saddle points whose reaction paths led to the wrong reactant-product pair, revealing local geometric intuition without a robust grasp of the global reaction coordinate. TSBench establishes a mechanism-level yardstick for LLM agents in mechanism-sensitive tasks such as synthesis planning and autonomous experimentation.
Related
- Mechanism-grounded fragmentation trees enable explicit chemical reasoning for electron ionization mass spectrum prediction
- When Do Quantum-Chemical Descriptors Improve Aqueous Solubility Prediction? A Residual-Rescue Benchmark under Chemical Distribution Shift
- Auditing LLM Chemical Priors for Multi-Objective Bayesian Optimization of Chemical Reactions: A Validity, Alignment, and Diversity Benchmark
- The Understanding of Size‐Dependent Magnetic Loss Mechanisms Based on Surface Atoms Moment to Tune Electromagnetic Wave Response
- Understanding oxygen reduction/evolution reaction and degradation mechanisms on BaCo0.4Fe0.4Zr0.1Y0.1O3-δ electrode in protonic ceramic electrochemical cells via the distribution of relaxation times analysis
BibTeX
@misc{xu2026tsbench,
title = {{TSBench: A physics-grounded benchmark for evaluating LLM understanding of chemical reaction mechanisms}},
author = {Xiaohu Xu and Tong Zhu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.08503},
eprint = {2609.08503},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.08503},
}
RIS
TY - UNPB
TI - TSBench: A physics-grounded benchmark for evaluating LLM understanding of chemical reaction mechanisms
AU - Xiaohu Xu
AU - Tong Zhu
PY - 2026
DA - 2026/09/08
PB - arXiv
DO - 10.48550/arxiv.2609.08503
UR - https://arxiv.org/abs/2609.08503
AB - Understanding a chemical reaction requires mapping a symbolic reactant-product description onto the three-dimensional pathway through which atoms rearrange, yet chemistry benchmarks for large language models (LLMs) largely probe factual knowledge and text-based reasoning. Here we introduce TSBench, a benchmark in which an LLM agent uses structure-editing tools to construct three-dimensional transition-state (TS) guesses verified by an automated quantum-chemical pipeline, yielding a physics-grounded pass/fail verdict. Across 546 evaluations of seven frontier LLMs on 78 elementary reactions, the aggregate success rate rose from 50.4% to 66.8% under diagnosis-driven revision; the best models approached 90% on the simplest reactions, yet performance dropped sharply with mechanistic complexity. Most failed attempts produced locally plausible saddle points whose reaction paths led to the wrong reactant-product pair, revealing local geometric intuition without a robust grasp of the global reaction coordinate. TSBench establishes a mechanism-level yardstick for LLM agents in mechanism-sensitive tasks such as synthesis planning and autonomous experimentation.
ER -