Evaluating and improving AI agents for sequencing protocol understanding
Abstract
Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.
Related
- Evaluating Autonomous LLM Agents Across Molecular Prediction and Optimization Benchmarks
- Evaluating AI agents on the IUCN Red List assessor exam: Tradeoffs and opportunities for AI assistants in extinction risk assessments
- Evaluating the efficacy and safety of therapeutic plasma exchange in snakebite-associated thrombotic microangiopathy compared to standard care alone: study protocol for a multicenter randomized controlled trial
- Evaluating imputation methods for determining North American honey bee (Apis mellifera) ancestry from low-pass whole-genome sequencing data
- Reading Between the Reads: A Decade On, Revisiting Sequencing Protocol Annotation in the Sequence Read Archive
BibTeX
@misc{poon2026evaluating,
title = {{Evaluating and improving AI agents for sequencing protocol understanding}},
author = {Chi-Lam Poon and Michael Goneos and Zezheng Song and Xi Chen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.37851},
eprint = {2610.37851},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.37851},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating and improving AI agents for sequencing protocol understanding
AU - Chi-Lam Poon
AU - Michael Goneos
AU - Zezheng Song
AU - Xi Chen
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.37851
UR - https://arxiv.org/abs/2610.37851
AB - Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.
ER -