BioSecBench-Function: A Verifiable Benchmark for Reasoning about Biological Function from Experimental Data
- 1LatchBio
Abstract
Inferring biological function from experimental data is central to understanding emerging pathogens and developing effective countermeasures, yet interpreting these data remains slow and expert-intensive. AI agents could help accelerate this process by reasoning across sequence, structural, and biophysical evidence. We present BioSecBench-Function, a verifiable benchmark for recovering biosecurity-relevant function from real biological data. The benchmark comprises 111 evaluations built from published datasets and graded deterministically against ground truth. We organize evaluations along two dimensions: threat axis (spanning seven biosecurity-relevant question types) and biological question (indicating whether the solution depends primarily on sequence, structure, or biophysical assay data). Across 7,326 runs from twenty-two model-harness configurations, Opus 5 under Claude Code led on endpoint pass rate at 50.3%, and Grok 4.6 under Grok Build led on overall pass rate at 44.1% when refusals counted as failures. Performance varied substantially across both model-harness configurations and task categories. Refusal rates differed sharply by provider, and cost was a poor predictor of accuracy: several configurations exceeded 40% pass rate at low cost. BioSecBench-Function provides a standard for measuring whether agents can be trusted to interpret what a new pathogen or variant does when the next outbreak arrives.
Related
- Microenvironment awareness pragmatic analysis for generative enzyme function reasoning
- Specialty-aware benchmark design, split integrity, and baseline evaluation for Korean medical question answering
- Bolek: A Multimodal Language Model for Molecular Reasoning
- DICD-IE: an expert-annotated benchmark for evaluating pretrained biomedical language models in dental implant literature and clinical knowledge extraction
- RNADyn: A Benchmark for Generating and Understanding RNA Dynamics
BibTeX
@misc{wang2026biosecbenchfunction,
title = {{BioSecBench-Function: A Verifiable Benchmark for Reasoning about Biological Function from Experimental Data}},
author = {D. Wang and Q. Xu and A. Banerjee and R. Jain and A. Vermani and J. Felix and G. Moreno and F. AlZaben and N. Keul and E. Seeyave and H. Bhasin},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.03.749010},
url = {https://www.biorxiv.org/content/10.64898/2026.09.03.749010v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - BioSecBench-Function: A Verifiable Benchmark for Reasoning about Biological Function from Experimental Data
AU - D. Wang
AU - Q. Xu
AU - A. Banerjee
AU - R. Jain
AU - A. Vermani
AU - J. Felix
AU - G. Moreno
AU - F. AlZaben
AU - N. Keul
AU - E. Seeyave
AU - H. Bhasin
PY - 2026
DA - 2026/09/03
PB - bioRxiv
DO - 10.64898/2026.09.03.749010
UR - https://www.biorxiv.org/content/10.64898/2026.09.03.749010v1
AB - Inferring biological function from experimental data is central to understanding emerging pathogens and developing effective countermeasures, yet interpreting these data remains slow and expert-intensive. AI agents could help accelerate this process by reasoning across sequence, structural, and biophysical evidence. We present BioSecBench-Function, a verifiable benchmark for recovering biosecurity-relevant function from real biological data. The benchmark comprises 111 evaluations built from published datasets and graded deterministically against ground truth. We organize evaluations along two dimensions: threat axis (spanning seven biosecurity-relevant question types) and biological question (indicating whether the solution depends primarily on sequence, structure, or biophysical assay data). Across 7,326 runs from twenty-two model-harness configurations, Opus 5 under Claude Code led on endpoint pass rate at 50.3%, and Grok 4.6 under Grok Build led on overall pass rate at 44.1% when refusals counted as failures. Performance varied substantially across both model-harness configurations and task categories. Refusal rates differed sharply by provider, and cost was a poor predictor of accuracy: several configurations exceeded 40% pass rate at low cost. BioSecBench-Function provides a standard for measuring whether agents can be trusted to interpret what a new pathogen or variant does when the next outbreak arrives.
ER -