Leakage-Controlled Benchmarking of Variant Effect Predictors and Development of Gradient-Boosting Consensus Model
- 1Loschmidt Laboratories, Department of Experimental Biology and RECETOX, Faculty of Science, Masaryk University, Kotlarska 2, Brno, Czech Republic
Abstract
Single amino acid mutations are among the most common forms of genetic variation, with effects ranging from negligible to severe disruption of protein function. Accurate quantitative evaluation of protein variants therefore represents a key challenge in medicine and protein engineering. Computational variant effect predictors (VEPs) are widely used to prioritize missense variants, yet their fair comparison is hindered by data circularity, benchmarks restricted to human clinical annotations, and limited availability of code and training data, all of which can inflate apparent performance. Here, we reviewed 18 recent and 3 older, widely adopted variant effect predictors for reproducible benchmarking and assembled a rigorously curated, leakage-controlled Benchmark dataset by integrating clinically annotated human variants from ClinVar and Humsavar as well as cross-species variants with functional annotations. The evaluation on this dataset revealed a narrow overall performance range for the benchmarked tools (AUC 0.631-0.758) and a consistent drop from clinical to functional data. Surprisingly, several tools published two decades ago demonstrated comparable performance to mid-ranked modern predictors. Building on these results, we developed a gradient-boosting consensus model, PredictSNPprot, which achieved an AUC of 0.82 on a held-out test set, outperforming the original PredictSNP meta-predictor and the best individual state-of-the-art tool by 13.6 and 5.7 percentage points, respectively. The Benchmark dataset, including the predictions from the evaluated tools, is available at https://doi.org/10.5281/zenodo.23597130 and PredictSNPprot consensus predictor is available at https://github.com/loschmidt/PredictSNPprot.
Related
- Leakage-controlled benchmarking reveals generalization limits of deep learning for protein-ligand binding affinity prediction
- Quantifying the reliability gap in cross-domain plant disease classification: benchmarking the limited efficacy of standard mitigation techniques under controlled-to-field shift
- Controlled gonad development: effects of environmental factors on sexual maturity in red swamp crayfish, Procambarus clarkii
- Reflections on the ESC consensus statement on inflammation in coronary syndromes: From mechanistic certainty to practical cardiology
- The Management of Type 1 Diabetes in Adults. The Updated 2026 Consensus Report by the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD)
BibTeX
@misc{havlasek2026leakagecontrolled,
title = {{Leakage-Controlled Benchmarking of Variant Effect Predictors and Development of Gradient-Boosting Consensus Model}},
author = {M. Havlasek and P. Kabourek and M. Musil and J. Damborsky and S. Mazurenko and D. Bednar},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.30.763022},
url = {https://www.biorxiv.org/content/10.64898/2026.10.30.763022v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Leakage-Controlled Benchmarking of Variant Effect Predictors and Development of Gradient-Boosting Consensus Model
AU - M. Havlasek
AU - P. Kabourek
AU - M. Musil
AU - J. Damborsky
AU - S. Mazurenko
AU - D. Bednar
PY - 2026
DA - 2026/10/30
PB - bioRxiv
DO - 10.64898/2026.10.30.763022
UR - https://www.biorxiv.org/content/10.64898/2026.10.30.763022v1
AB - Single amino acid mutations are among the most common forms of genetic variation, with effects ranging from negligible to severe disruption of protein function. Accurate quantitative evaluation of protein variants therefore represents a key challenge in medicine and protein engineering. Computational variant effect predictors (VEPs) are widely used to prioritize missense variants, yet their fair comparison is hindered by data circularity, benchmarks restricted to human clinical annotations, and limited availability of code and training data, all of which can inflate apparent performance. Here, we reviewed 18 recent and 3 older, widely adopted variant effect predictors for reproducible benchmarking and assembled a rigorously curated, leakage-controlled Benchmark dataset by integrating clinically annotated human variants from ClinVar and Humsavar as well as cross-species variants with functional annotations. The evaluation on this dataset revealed a narrow overall performance range for the benchmarked tools (AUC 0.631-0.758) and a consistent drop from clinical to functional data. Surprisingly, several tools published two decades ago demonstrated comparable performance to mid-ranked modern predictors. Building on these results, we developed a gradient-boosting consensus model, PredictSNPprot, which achieved an AUC of 0.82 on a held-out test set, outperforming the original PredictSNP meta-predictor and the best individual state-of-the-art tool by 13.6 and 5.7 percentage points, respectively. The Benchmark dataset, including the predictions from the evaluated tools, is available at https://doi.org/10.5281/zenodo.23597130 and PredictSNPprot consensus predictor is available at https://github.com/loschmidt/PredictSNPprot.
ER -