Evaluating Instrument-Centered AI Agents for Table-Top Science Experiments

Abstract

Reliable evaluation of AI agents for table-top science experiments has remained elusive because live instruments are hard to reset and share at scale, while instrument statefulness and safety constraints complicate evaluation. We introduce instrMCP and q-devBench, an evaluation stack for instrument-centered AI agents in table-top experiments. instrMCP connects coding agents to scientific instruments through the Model Context Protocol, serving as the instrument-access harness. q-devBench reconstructs 37 real experimental tasks from condensed-matter physics as virtual QCoDeS instruments covering quantum electronic transport, scanning probe imaging, and calibration workflows. q-devBench exposes reproducible failure modes in laboratory-agent systems while preserving realistic instrument-control workflows. Across a range of commercial coding-agent configurations, current systems already show substantial operational competence, satisfying 70-83% of measurement-related rubric criteria. The main bottleneck is scientific analysis, where performance falls to 28-46%. In an analysis-only ablation in which agents receive clean reference datasets without running the virtual measurement session, analysis performance improves substantially, closing roughly two thirds of the difference between the measurement and analysis scores. This suggests that much of the deficit arises from the measurement-session context, including tool-use overhead, longer context, measurement-planning decisions, and possible acquisition errors, while a residual scientific-reasoning gap remains. Together, instrMCP and q-devBench provide a practical sandbox for benchmarking, diagnosing, and improving laboratory AI harnesses, and establish a route from controlled virtual evaluation to future live-hardware validation.

HARMONI@ELT: instrument design, performance, and science capabilities
B Neichel, J Dunlop, S Chittick +16
HAL1 Sep 2026PreprintLicence not statedGeneral astronomy Applied physics & instrumentationHAL
The unified theory at eleven parameters and the experiments: instrument by instrument, what is tested and what would refute
Patrick B. Reynaud
HAL26 Sep 2026PreprintCC BY 4.0Full textGeneral physicsHAL
Optimising instrument concepts with Machine Learning: Application to CMB spectral distortion experiments
Xavier Coulon, Nabila Aghanim, Bruno Maffei
HAL15 Sep 2026PreprintCC BY 4.0Full textCosmology Instrumentation & methodsHALarXiv
NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents
Lijie Ding, Changwoo Do
arXiv2 Oct 2026PreprintarXiv licenceFull textApplied physics & instrumentationarXiv
Evaluating the Reliability of Cross-Instrument Comparison: (p)XRF Measurements of Two Rare 'Kangxi-Reign' Enameled Glasses
Gulsu Simsek-Franci, Philippe Colomban, Tung-Ho Chen +4
Ceramics11 Sep 2026Peer-reviewedCC BY 4.0Full textAnalytical chemistry Condensed matterHAL

Cite

BibTeX RIS

BibTeX

@misc{cai2026evaluating,
  title = {{Evaluating Instrument-Centered AI Agents for Table-Top Science Experiments}},
  author = {Jiaqi Cai and Siyu Cheng and William S. Huxter and Li-Qiao Xia and Shuwen Sun and Shuai Yuan and Yifan Zhao and Naiyuan Zhang and Weijie Li and Muxian Xu and Ilija Zeljkovic and Xiaodong Xu and Pablo Jarillo-Herrero},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.41717},
  eprint = {2610.41717},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41717},
}

RIS

TY  - UNPB
TI  - Evaluating Instrument-Centered AI Agents for Table-Top Science Experiments
AU  - Jiaqi Cai
AU  - Siyu Cheng
AU  - William S. Huxter
AU  - Li-Qiao Xia
AU  - Shuwen Sun
AU  - Shuai Yuan
AU  - Yifan Zhao
AU  - Naiyuan Zhang
AU  - Weijie Li
AU  - Muxian Xu
AU  - Ilija Zeljkovic
AU  - Xiaodong Xu
AU  - Pablo Jarillo-Herrero
PY  - 2026
DA  - 2026/10/30
PB  - arXiv
DO  - 10.48550/arxiv.2610.41717
UR  - https://arxiv.org/abs/2610.41717
AB  - Reliable evaluation of AI agents for table-top science experiments has remained elusive because live instruments are hard to reset and share at scale, while instrument statefulness and safety constraints complicate evaluation. We introduce instrMCP and q-devBench, an evaluation stack for instrument-centered AI agents in table-top experiments. instrMCP connects coding agents to scientific instruments through the Model Context Protocol, serving as the instrument-access harness. q-devBench reconstructs 37 real experimental tasks from condensed-matter physics as virtual QCoDeS instruments covering quantum electronic transport, scanning probe imaging, and calibration workflows. q-devBench exposes reproducible failure modes in laboratory-agent systems while preserving realistic instrument-control workflows. Across a range of commercial coding-agent configurations, current systems already show substantial operational competence, satisfying 70-83% of measurement-related rubric criteria. The main bottleneck is scientific analysis, where performance falls to 28-46%. In an analysis-only ablation in which agents receive clean reference datasets without running the virtual measurement session, analysis performance improves substantially, closing roughly two thirds of the difference between the measurement and analysis scores. This suggests that much of the deficit arises from the measurement-session context, including tool-use overhead, longer context, measurement-planning decisions, and possible acquisition errors, while a residual scientific-reasoning gap remains. Together, instrMCP and q-devBench provide a practical sandbox for benchmarking, diagnosing, and improving laboratory AI harnesses, and establish a route from controlled virtual evaluation to future live-hardware validation.
ER  -

Abstract

Related

Cite