Benchmarking virus identification tools on vaginal metagenomes
- 1Department of Medical Biochemistry and Microbiology, Science for Life Laboratory, Uppsala University, Uppsala, Sweden
Abstract
The vaginal microbiome is associated with reproductive and sexual health, yet its viral component remains understudied. A recent vaginal genome catalogue reported that 85.8% of its species-level viral groups were absent from all five viral databases examined. Viral identification tools depend on reference sequences, either directly or through their training data. Previous benchmarks used simulated or non-vaginal datasets. How well these tools generalise to vaginal metagenomes therefore remains unclear. We benchmarked 14 tools spanning five methodological approaches. Evaluations used controlled genome fragments, viral spike-in co-assemblies and virus-like particle (VLP)-enriched co-assemblies. A multi-evidence benchmark assessed 13 vaginal shotgun assemblies (primary cohort). An independent 30-sample cohort provided external validation. geNomad ranked first or second by Matthews correlation coefficient (MCC) in every ranked evaluation. VIBRANT achieved similar MCC to geNomad on the primary assemblies. It made eight false-positive calls among 11,577 negatives but recovered only 22% of viral contigs. VIBRANT ranked first in the independent cohort. Tool rankings agreed closely between cohorts (Spearman rho = 0.952; 95% CI 0.921 - 0.960). Detection of sequences lacking reference homologues differed substantially between tools. Two phage genomes without detectable reference homologues provided a controlled novelty test. Each evaluated composition-based tool recovered at least 74% of their 1,500-bp fragments. Marker- or reference-dependent tools recovered at most 31%. A separate analysis examined 48 unconfirmed candidate viral contigs lacking detectable nucleotide or protein homologues. Jaeger flagged 41, compared with four for VIBRANT and one for geNomad. On short fragments, attention-based tools achieved 4.0-5.5 times the mean area under the precision-recall curve of feature-based tools. ViraLM accounted for most of this difference. ViraLM also had the highest recall (sensitivity) of eukaryote-infecting viruses in the evaluated panel. VirSorter2 achieved the highest MCC on VLP-enriched contigs. These results support geNomad when prioritising overall classification performance and VIBRANT when minimising false-positive phage calls is the priority. ViraLM supports short-contig and eukaryote-infecting-virus detection. VirSorter2 supports identification in VLP-enriched assemblies, and Jaeger supports screening for candidate novel phages. These findings guide tool selection for studies investigating the vaginal virome in reproductive and sexual health.
Related
- Benchmarking and integration of viral prediction tools to uncover viral dark matter across divergent low abundance metagenomes
- Benchmarking strain-level profiling of Escherichia coli in short-read gut metagenomes
- Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
- Benchmarking short-read germline structural variant calling highlights advantages of using ensembles of tools and small impact of graph genome alignment
- Benchmarking circRNA Detection Tools from Long-Read Sequencing Using Data-Driven and Flexible Simulation Framework
BibTeX
@misc{dube2026benchmarking,
title = {{Benchmarking virus identification tools on vaginal metagenomes}},
author = {F. Dube and A.-U. Happel and H. B. Jaspan and L. W. Hugerth},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.24.753817},
url = {https://www.biorxiv.org/content/10.64898/2026.09.24.753817v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Benchmarking virus identification tools on vaginal metagenomes
AU - F. Dube
AU - A.-U. Happel
AU - H. B. Jaspan
AU - L. W. Hugerth
PY - 2026
DA - 2026/09/24
PB - bioRxiv
DO - 10.64898/2026.09.24.753817
UR - https://www.biorxiv.org/content/10.64898/2026.09.24.753817v1
AB - The vaginal microbiome is associated with reproductive and sexual health, yet its viral component remains understudied. A recent vaginal genome catalogue reported that 85.8% of its species-level viral groups were absent from all five viral databases examined. Viral identification tools depend on reference sequences, either directly or through their training data. Previous benchmarks used simulated or non-vaginal datasets. How well these tools generalise to vaginal metagenomes therefore remains unclear. We benchmarked 14 tools spanning five methodological approaches. Evaluations used controlled genome fragments, viral spike-in co-assemblies and virus-like particle (VLP)-enriched co-assemblies. A multi-evidence benchmark assessed 13 vaginal shotgun assemblies (primary cohort). An independent 30-sample cohort provided external validation. geNomad ranked first or second by Matthews correlation coefficient (MCC) in every ranked evaluation. VIBRANT achieved similar MCC to geNomad on the primary assemblies. It made eight false-positive calls among 11,577 negatives but recovered only 22% of viral contigs. VIBRANT ranked first in the independent cohort. Tool rankings agreed closely between cohorts (Spearman rho = 0.952; 95% CI 0.921 - 0.960). Detection of sequences lacking reference homologues differed substantially between tools. Two phage genomes without detectable reference homologues provided a controlled novelty test. Each evaluated composition-based tool recovered at least 74% of their 1,500-bp fragments. Marker- or reference-dependent tools recovered at most 31%. A separate analysis examined 48 unconfirmed candidate viral contigs lacking detectable nucleotide or protein homologues. Jaeger flagged 41, compared with four for VIBRANT and one for geNomad. On short fragments, attention-based tools achieved 4.0-5.5 times the mean area under the precision-recall curve of feature-based tools. ViraLM accounted for most of this difference. ViraLM also had the highest recall (sensitivity) of eukaryote-infecting viruses in the evaluated panel. VirSorter2 achieved the highest MCC on VLP-enriched contigs. These results support geNomad when prioritising overall classification performance and VIBRANT when minimising false-positive phage calls is the priority. ViraLM supports short-contig and eukaryote-infecting-virus detection. VirSorter2 supports identification in VLP-enriched assemblies, and Jaeger supports screening for candidate novel phages. These findings guide tool selection for studies investigating the vaginal virome in reproductive and sexual health.
ER -