VariantFlow: a selective-execution engine for efficient population genomic computation on large variant datasets
- 1Umeå Plant Science Centre, Department of Plant Physiology, Umeå University, SE-901 87 Umeå, Sweden
Abstract
Population-scale sequencing now produces variant call sets with thousands of samples and millions of sites, making post-calling analysis a recurring bottleneck. Because the Variant Call Format stores every field of every record together, a tool answering a field-limited question still parses the unused annotations, FORMAT blocks, and per-sample values, which costs time without changing the result. We present VariantFlow, a command-line engine built on selective execution, decoding only the fields the current filter, statistic, or projection requires and preserving original records where possible. On correctness-matched benchmarks on the 1000 Genomes 3,202-sample high-coverage dataset, VariantFlow computed whole-chromosome missingness 8.0-8.5x faster than VCFtools at a constant 8.6 MB of memory, the chromosome 1 summary taking 128 s against 1031 s. An index-assisted FILTER=PASS predicate ran 123-273x faster than bcftools where block metadata excludes most of the file. Further gains, reported in Table 1, cover FORMAT-rich filtering on record subsets, linkage disequilibrium, and export-once Parquet queries served through DuckDB. Core population-genetic outputs were byte-identical to VCFtools, per-individual missingness and the site-frequency spectrum matched scikit-allel exactly, and the missing-data-aware pi and dxy estimators reproduced pixy's pairwise counts in every window. By decoding only the fields each operation requires, VariantFlow brings large-cohort post-calling analysis within reach of exploratory work on a single workstation. It complements rather than replaces bcftools, HTSlib, GATK, and VCFtools, and should be of value wherever large cohort VCFs are repeatedly summarised, filtered, or exported. VariantFlow is open source (Rust, MIT OR Apache-2.0) at https://github.com/ehsanestaji/VariantFlow; version 1.5.0 is archived at doi:10.5281/zenodo.21198172.
Related
- Structure alone supports efficient visual computation in the Drosophila visual system
- Population density is a selective driver of urban soil microbiomes: evidence from bacteria, fungi and protozoan communities in a regional city
- Evidence of Selective Response in the Free-Breeding Dog Population in Chornobyl.
- QuImputer: an LD-informed QUBO/Ising formulation of genotype imputation with quantum-hardware execution
- The GA4GH Workflow Execution Service: A Standard API for Workflow Execution and Management in Genomics
BibTeX
@misc{estaji2026variantflow,
title = {{VariantFlow: a selective-execution engine for efficient population genomic computation on large variant datasets}},
author = {E. Estaji and J.-F. Mao},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.01.748643},
url = {https://www.biorxiv.org/content/10.64898/2026.09.01.748643v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - VariantFlow: a selective-execution engine for efficient population genomic computation on large variant datasets
AU - E. Estaji
AU - J.-F. Mao
PY - 2026
DA - 2026/09/01
PB - bioRxiv
DO - 10.64898/2026.09.01.748643
UR - https://www.biorxiv.org/content/10.64898/2026.09.01.748643v1
AB - Population-scale sequencing now produces variant call sets with thousands of samples and millions of sites, making post-calling analysis a recurring bottleneck. Because the Variant Call Format stores every field of every record together, a tool answering a field-limited question still parses the unused annotations, FORMAT blocks, and per-sample values, which costs time without changing the result. We present VariantFlow, a command-line engine built on selective execution, decoding only the fields the current filter, statistic, or projection requires and preserving original records where possible. On correctness-matched benchmarks on the 1000 Genomes 3,202-sample high-coverage dataset, VariantFlow computed whole-chromosome missingness 8.0-8.5x faster than VCFtools at a constant 8.6 MB of memory, the chromosome 1 summary taking 128 s against 1031 s. An index-assisted FILTER=PASS predicate ran 123-273x faster than bcftools where block metadata excludes most of the file. Further gains, reported in Table 1, cover FORMAT-rich filtering on record subsets, linkage disequilibrium, and export-once Parquet queries served through DuckDB. Core population-genetic outputs were byte-identical to VCFtools, per-individual missingness and the site-frequency spectrum matched scikit-allel exactly, and the missing-data-aware pi and dxy estimators reproduced pixy's pairwise counts in every window. By decoding only the fields each operation requires, VariantFlow brings large-cohort post-calling analysis within reach of exploratory work on a single workstation. It complements rather than replaces bcftools, HTSlib, GATK, and VCFtools, and should be of value wherever large cohort VCFs are repeatedly summarised, filtered, or exported. VariantFlow is open source (Rust, MIT OR Apache-2.0) at https://github.com/ehsanestaji/VariantFlow; version 1.5.0 is archived at doi:10.5281/zenodo.21198172.
ER -