RECOB: Reliable Benchmarking of Experimental Optimization in Chemistry and Materials Science
Abstract
Optimization methods for experimental science are often evaluated on synthetic functions that are reproducible but omit important characteristics of real experiments. We introduce RECOB (REliable Chem Optimization Benchmark, Github repository: \hyperlink{https://github.com/XieZikai/RECOB}{https://github.com/XieZikai/RECOB}), a black-box optimization benchmark constructed exclusively from data generated through physical experiments in chemistry and materials science. The suite contains 14 single-objective and two multi-objective tasks spanning chemical reactions, material formulations, electrochemical systems, continuous-flow processes, and automated laboratories. Each task provides a machine-readable specification of its decision variables, feasible domain, physical constraints, objective direction, and experimental provenance. Continuously queryable learned oracles are screened using repeated holdout validation and prespecified admission criteria, while measured-table replay enables evaluation using the original experimental responses. Under a common paired evaluation protocol, we compare ten single-objective and eight multi-objective optimization methods. HEBO achieves the best aggregate single-objective rank, while qNEHVI leads the multi-objective comparison. Model-based methods generally outperform non-adaptive baselines, although their computational overhead varies substantially. We further assess benchmark reliability using independent oracle retraining and measured-table replay. Aggregate optimizer rankings remain highly consistent across retrained oracles, while replay preserves the broad performance hierarchy using only physically measured responses. Together, these results show that RECOB can reproducibly distinguish optimizer performance as an experimentally grounded and reliability-tested benchmark for black-box optimization in chemistry and materials science.
Related
- Chemistry-Informed Transfer and Interaction-Resolved Learning Towards Data-Efficient Materials Exploration
- From unstructured to structured: opportunities, risks and engineering practices for LLM-based data extraction in chemistry and materials
- Graph Symbolic Distillation: Decoding Graph Data into Interpretable Formulas across Materials and Chemistry
- Measurement-efficient energy estimator for quantum chemistry of photodynamic materials
- GVIM 2.0: A Multi-Agent Research System for Chemistry and Materials Science
BibTeX
@misc{xie2026recob,
title = {{RECOB: Reliable Benchmarking of Experimental Optimization in Chemistry and Materials Science}},
author = {Zikai Xie and Jiaming Wan and Linjiang Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.20891},
eprint = {2609.20891},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.20891},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - RECOB: Reliable Benchmarking of Experimental Optimization in Chemistry and Materials Science
AU - Zikai Xie
AU - Jiaming Wan
AU - Linjiang Chen
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2609.20891
UR - https://arxiv.org/abs/2609.20891
AB - Optimization methods for experimental science are often evaluated on synthetic functions that are reproducible but omit important characteristics of real experiments. We introduce RECOB (REliable Chem Optimization Benchmark, Github repository: \hyperlink{https://github.com/XieZikai/RECOB}{https://github.com/XieZikai/RECOB}), a black-box optimization benchmark constructed exclusively from data generated through physical experiments in chemistry and materials science. The suite contains 14 single-objective and two multi-objective tasks spanning chemical reactions, material formulations, electrochemical systems, continuous-flow processes, and automated laboratories. Each task provides a machine-readable specification of its decision variables, feasible domain, physical constraints, objective direction, and experimental provenance. Continuously queryable learned oracles are screened using repeated holdout validation and prespecified admission criteria, while measured-table replay enables evaluation using the original experimental responses. Under a common paired evaluation protocol, we compare ten single-objective and eight multi-objective optimization methods. HEBO achieves the best aggregate single-objective rank, while qNEHVI leads the multi-objective comparison. Model-based methods generally outperform non-adaptive baselines, although their computational overhead varies substantially. We further assess benchmark reliability using independent oracle retraining and measured-table replay. Aggregate optimizer rankings remain highly consistent across retrained oracles, while replay preserves the broad performance hierarchy using only physically measured responses. Together, these results show that RECOB can reproducibly distinguish optimizer performance as an experimentally grounded and reliability-tested benchmark for black-box optimization in chemistry and materials science.
ER -