Genetic Perturbation Prediction Models Fail Out-of-Distribution: A Rigorous Benchmark and Systematic Evaluation
- 1Elucidata Corporation
Abstract
Perturbation prediction models - which forecast transcriptional responses to gene inactivation - are increasingly proposed as in silico surrogates for genome-scale drug discovery screens. Yet, rigorous evaluation of whether these models genuinely generalize beyond their training distribution has been lacking. Existing benchmarks rely predominantly on random train-test splits, obscuring systematic failure modes that only emerge under realistic out-of-distribution (OOD) conditions. Here we introduce EL-PERTURB (Extrapolative Learning for PERTURBation Prediction), a benchmarking framework that stress-tests 20 perturbation prediction models spanning six architecture classes on both essential gene and genome-wide Perturb-seq screens, using a novel stratified OOD splitting strategy that partitions train and test sets into functionally distinct regions of transcriptional space. We find that model performance degrades substantially under OOD conditions, and that architectural complexity - including single-cell foundation model embeddings - confers no reliable advantage over well-tuned simpler baselines. EL-PERTURB establishes a rigorous benchmark for perturbation effect prediction across a gradient of generalization difficulty, providing a guide for building models that can genuinely serve as surrogates for wet-lab screening.
Related
- PopPert: Population-level Joint-Distribution Modeling for Single-Cell Perturbation Prediction
- SAGE: A sampling-aware global evaluation benchmark for species distribution modeling
- Where single-cell foundation models help: a benchmark of annotation, perturbation and regulatory inference against simple baselines
- Centered cross-context perturbation transfer on the public 2025 Virtual Cell Challenge benchmark
- PerturbBridge: Conditional Latent Schr\"{o}dinger Bridge for Single-Cell Perturbation Prediction
BibTeX
@misc{tiwari2026genetic,
title = {{Genetic Perturbation Prediction Models Fail Out-of-Distribution: A Rigorous Benchmark and Systematic Evaluation}},
author = {K. Tiwari and G. Mahajan and R. P. Bellad and N. K. Dhruw and A. K. Jha},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.16.759604},
url = {https://www.biorxiv.org/content/10.64898/2026.10.16.759604v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Genetic Perturbation Prediction Models Fail Out-of-Distribution: A Rigorous Benchmark and Systematic Evaluation
AU - K. Tiwari
AU - G. Mahajan
AU - R. P. Bellad
AU - N. K. Dhruw
AU - A. K. Jha
PY - 2026
DA - 2026/10/16
PB - bioRxiv
DO - 10.64898/2026.10.16.759604
UR - https://www.biorxiv.org/content/10.64898/2026.10.16.759604v1
AB - Perturbation prediction models - which forecast transcriptional responses to gene inactivation - are increasingly proposed as in silico surrogates for genome-scale drug discovery screens. Yet, rigorous evaluation of whether these models genuinely generalize beyond their training distribution has been lacking. Existing benchmarks rely predominantly on random train-test splits, obscuring systematic failure modes that only emerge under realistic out-of-distribution (OOD) conditions. Here we introduce EL-PERTURB (Extrapolative Learning for PERTURBation Prediction), a benchmarking framework that stress-tests 20 perturbation prediction models spanning six architecture classes on both essential gene and genome-wide Perturb-seq screens, using a novel stratified OOD splitting strategy that partitions train and test sets into functionally distinct regions of transcriptional space. We find that model performance degrades substantially under OOD conditions, and that architectural complexity - including single-cell foundation model embeddings - confers no reliable advantage over well-tuned simpler baselines. EL-PERTURB establishes a rigorous benchmark for perturbation effect prediction across a gradient of generalization difficulty, providing a guide for building models that can genuinely serve as surrogates for wet-lab screening.
ER -