MetagenomicsBench: A Verifiable Benchmark for Agentic Metagenomics Analysis
- 1LatchBio
Abstract
Metagenomics has transformed our ability to characterize microbial communities without relying on cultivation, but translating complex microbiome datasets into reliable biological conclusions remains a major analytical challenge. Although AI agents have improved substantially at software engineering and general data analysis, it remains unclear whether they can reliably analyze and interpret real-world metagenomic data. We introduce MetagenomicsBench, a benchmark of 100 verifiable evaluations derived from published datasets spanning community structure, host-microbiome associations, microbial function, longitudinal dynamics, and microbiome interventions. Each evaluation provides experimental data and a deterministic grader that evaluates recovery of a key analytical or biological result. Across 10,200 trajectories from 34 model-harness configurations, the strongest configuration achieved a 60.0% pass rate. Performance varied across functional domains, data modalities, and biological systems, while greater cost, token usage, and tool use did not consistently correspond to higher accuracy. Trajectory analysis showed that agents often produced technically plausible analyses while making mistakes in problem interpretation, statistical reasoning, and biological interpretation. MetagenomicsBench provides a framework for measuring progress toward AI agents that can reliably make and validate the analytical and biological decisions required for metagenomic research.
Related
- Verifiable Agentic Science on Public Infrastructure
- S2F-Agent: Harnessing sequence-to-function models for verifiable genome interpretation
- Genome-resolved metagenomics reveals the taxonomic and functional landscape of the rumen microbiome in Japanese Black × Holstein cattle and its associations with carcass traits
- Whole-Genome Metagenomics Insights Revealed a Uranium Bio-remediating Cross-Domain Microbiome in the Dhala Impact Structure, India
- Longitudinal strain-level metagenomics reveals reciprocal Lactobacillus dynamics and outcome-associated protein variants during IVF treatment
BibTeX
@misc{yang2026metagenomicsbench,
title = {{MetagenomicsBench: A Verifiable Benchmark for Agentic Metagenomics Analysis}},
author = {Z. Yang and J. Connolly and A. Banerjee and S. Madad},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.05.756819},
url = {https://www.biorxiv.org/content/10.64898/2026.10.05.756819v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - MetagenomicsBench: A Verifiable Benchmark for Agentic Metagenomics Analysis
AU - Z. Yang
AU - J. Connolly
AU - A. Banerjee
AU - S. Madad
PY - 2026
DA - 2026/10/05
PB - bioRxiv
DO - 10.64898/2026.10.05.756819
UR - https://www.biorxiv.org/content/10.64898/2026.10.05.756819v1
AB - Metagenomics has transformed our ability to characterize microbial communities without relying on cultivation, but translating complex microbiome datasets into reliable biological conclusions remains a major analytical challenge. Although AI agents have improved substantially at software engineering and general data analysis, it remains unclear whether they can reliably analyze and interpret real-world metagenomic data. We introduce MetagenomicsBench, a benchmark of 100 verifiable evaluations derived from published datasets spanning community structure, host-microbiome associations, microbial function, longitudinal dynamics, and microbiome interventions. Each evaluation provides experimental data and a deterministic grader that evaluates recovery of a key analytical or biological result. Across 10,200 trajectories from 34 model-harness configurations, the strongest configuration achieved a 60.0% pass rate. Performance varied across functional domains, data modalities, and biological systems, while greater cost, token usage, and tool use did not consistently correspond to higher accuracy. Trajectory analysis showed that agents often produced technically plausible analyses while making mistakes in problem interpretation, statistical reasoning, and biological interpretation. MetagenomicsBench provides a framework for measuring progress toward AI agents that can reliably make and validate the analytical and biological decisions required for metagenomic research.
ER -