Structural Variant benchmarking frameworks: Parameterization, matching logic, and evaluation assumptions presented through HG002 and NA12878
- 1Department of Computer Engineering, Faculty of Computer and Informatics Engineering, Istanbul Technical University, Istanbul, Türkiye
- 2Department of Computer Engineering, Turkish Air Force Academy, National Defence University, Istanbul, Türkiye
Abstract
Structural Variants (SVs) constitute a significant yet complicated class of genomic changes. Due to their imprecise breakpoints, variable size, genomic context and representations across SV callers complicate their accurate detection. Several SV callers have been developed over the years; however, the process of evaluating the outputs of SV callers is inherently more complicated than single nucleotide polymorphisms (SNPs). Unlike SNPs where there is simply a binary representation of whether the nucleotide at a specific position exists, SV analysis must clarify partially overlapping events caused by imprecise breakpoints. This ambiguity has motivated the development of specialized frameworks to support the comparison of SVs. Each of these frameworks implement different assumptions about breakpoint resolution, size concordance or sequence similarity. In this study, we described the underlying strategy of SV comparison modules of three frameworks: Truvari, EvalSVcallers, and SVbenchmark. The primary focus of our study is to reveal how the implemented parameters of each framework influence matching behavior as well as their evaluation outcomes. The conceptual and algorithmic differences of these frameworks are further illustrated through an empirical example based on the query sets of five callers (Manta, Delly, Lumpy, GRIDSS, Wham) for the HG002 and NA12878 reference samples to stress how distinct benchmarking strategies translate into framework-dependent evaluation outcomes. Thus, the reported metrics demonstrate effects of such differences on reproducibility and interpretation rather than prioritizing performance ranking. Author summary: Structural Variants are large-scale DNA changes that can affect diversity and diseases; however, evaluating how accurately these variants are detected is not always straightforward. Different benchmarking frameworks use different matching rules when deciding whether a predicted variant matches a truth variant. Therefore, the same variant call set can receive different performance scores depending on the benchmarking framework and parameter settings used. This study compares three commonly used Structural Variant benchmarking frameworks namely, Truvari, EvalSVcallers, and SVbenchmark based on results from five different variant callers and two well-characterized human reference samples, HG002 and NA12878. The analyses show that both framework selection and parameter settings can alter precision, recall, F1-score, and the classification of individual variants. Additional resampling analyses show that most of these differences are not due to random changes in the evaluated genomic regions or variants, but are rather consistently stable. Variant level comparisons also reveal that discrepancies between frameworks are often related to differences in breakpoint location, variant size, or reciprocal interval overlap, but these features do not explain all discrepancies. These findings highlight the importance of transparent reporting of benchmarking methods and interpretation of performance metrics within the context of the framework used.
Related
- Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
- Benchmarking Large Language Models for Transcriptomic Data Analysis and Visualization: Multi-Model Workflow Evaluation
- The Genome In A Bottle HG002 assembly-based variant benchmark set enables comprehensive benchmarking of small and structural variants.
- Systematic evaluation of structural connectome thresholding in whole-brain network modelling
- A structural antibody benchmark for leakage-aware deep-learning evaluation
BibTeX
@article{maden2026structural,
title = {{Structural Variant benchmarking frameworks: Parameterization, matching logic, and evaluation assumptions presented through HG002 and NA12878}},
author = {Gamze Maden and Mehmet Baysan and Nizamettin Aydın},
journal = {PLOS Computational Biology},
year = {2026},
month = sep,
doi = {10.1371/journal.pcbi.1014824},
url = {https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1014824},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Structural Variant benchmarking frameworks: Parameterization, matching logic, and evaluation assumptions presented through HG002 and NA12878
AU - Gamze Maden
AU - Mehmet Baysan
AU - Nizamettin Aydın
PY - 2026
DA - 2026/09/23
JO - PLOS Computational Biology
DO - 10.1371/journal.pcbi.1014824
UR - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1014824
AB - Structural Variants (SVs) constitute a significant yet complicated class of genomic changes. Due to their imprecise breakpoints, variable size, genomic context and representations across SV callers complicate their accurate detection. Several SV callers have been developed over the years; however, the process of evaluating the outputs of SV callers is inherently more complicated than single nucleotide polymorphisms (SNPs). Unlike SNPs where there is simply a binary representation of whether the nucleotide at a specific position exists, SV analysis must clarify partially overlapping events caused by imprecise breakpoints. This ambiguity has motivated the development of specialized frameworks to support the comparison of SVs. Each of these frameworks implement different assumptions about breakpoint resolution, size concordance or sequence similarity. In this study, we described the underlying strategy of SV comparison modules of three frameworks: Truvari, EvalSVcallers, and SVbenchmark. The primary focus of our study is to reveal how the implemented parameters of each framework influence matching behavior as well as their evaluation outcomes. The conceptual and algorithmic differences of these frameworks are further illustrated through an empirical example based on the query sets of five callers (Manta, Delly, Lumpy, GRIDSS, Wham) for the HG002 and NA12878 reference samples to stress how distinct benchmarking strategies translate into framework-dependent evaluation outcomes. Thus, the reported metrics demonstrate effects of such differences on reproducibility and interpretation rather than prioritizing performance ranking. Author summary: Structural Variants are large-scale DNA changes that can affect diversity and diseases; however, evaluating how accurately these variants are detected is not always straightforward. Different benchmarking frameworks use different matching rules when deciding whether a predicted variant matches a truth variant. Therefore, the same variant call set can receive different performance scores depending on the benchmarking framework and parameter settings used. This study compares three commonly used Structural Variant benchmarking frameworks namely, Truvari, EvalSVcallers, and SVbenchmark based on results from five different variant callers and two well-characterized human reference samples, HG002 and NA12878. The analyses show that both framework selection and parameter settings can alter precision, recall, F1-score, and the classification of individual variants. Additional resampling analyses show that most of these differences are not due to random changes in the evaluated genomic regions or variants, but are rather consistently stable. Variant level comparisons also reveal that discrepancies between frameworks are often related to differences in breakpoint location, variant size, or reciprocal interval overlap, but these features do not explain all discrepancies. These findings highlight the importance of transparent reporting of benchmarking methods and interpretation of performance metrics within the context of the framework used.
SN - 1553-7358
ER -