A Pitfall in the Evaluation of Gene Perturbation Models
Abstract
Gene perturbation models aim to predict transcriptional responses to unseen genetic interventions, but comparisons rely on heterogeneous evaluation metrics and differentially-expressed (DE) gene subsets. We investigate the consequences of these choices using eight methods across six single-gene perturbation datasets under a unified evaluation protocol. Varying the number of top DE genes reveals substantial instability in Precision-based rankings, with the winning method changing 86.7% of the time, while rankings based on the rest of the metrics remain comparatively stable. Investigating this discrepancy exposes that repeating a single predicted expression profile across cell-level rows, a standard practice for many methods, can inflate significance estimates and produce excessively large predicted significant-DE sets. Specifically we found that 67.1% of tested perturbation--gene pairs pass the significance threshold despite some of them having minuscule effects. We derive an inflation factor showing that replicated predictions can amplify the standardized Mann--Whitney statistic even after correction for tied ranks, without adding independent information. This artifact changes Precision mainly as the number of DE genes increases because for smaller cutoffs, most of the genes are already significant due to the effect-based ordering. Taking this pitfall into account in our benchmark comparison, we find that a random forest using Gene Ontology features achieves the best overall results.
Related
- AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses
- Equivalence of Extensions and the Perturbation Index
- An Improved Upper Bound for the Perturbation Index
- Existence of multi-solitons for logarithmic Schrödinger equation with a perturbation of a smooth L2-subcritical function nonlinearity
- Pointwise Structures and Background Perturbation Analyses for Thermal Non-equilibrium Irrotational Flows in Three Space Dimensions
BibTeX
@misc{panagopoulos2026pitfall,
title = {{A Pitfall in the Evaluation of Gene Perturbation Models}},
author = {George Panagopoulos and Saman Omidi and Tsz Pan Tong and Jun Pang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.26267},
eprint = {2610.26267},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.26267},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Pitfall in the Evaluation of Gene Perturbation Models
AU - George Panagopoulos
AU - Saman Omidi
AU - Tsz Pan Tong
AU - Jun Pang
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2610.26267
UR - https://arxiv.org/abs/2610.26267
AB - Gene perturbation models aim to predict transcriptional responses to unseen genetic interventions, but comparisons rely on heterogeneous evaluation metrics and differentially-expressed (DE) gene subsets. We investigate the consequences of these choices using eight methods across six single-gene perturbation datasets under a unified evaluation protocol. Varying the number of top DE genes reveals substantial instability in Precision-based rankings, with the winning method changing 86.7% of the time, while rankings based on the rest of the metrics remain comparatively stable. Investigating this discrepancy exposes that repeating a single predicted expression profile across cell-level rows, a standard practice for many methods, can inflate significance estimates and produce excessively large predicted significant-DE sets. Specifically we found that 67.1% of tested perturbation--gene pairs pass the significance threshold despite some of them having minuscule effects. We derive an inflation factor showing that replicated predictions can amplify the standardized Mann--Whitney statistic even after correction for tied ranks, without adding independent information. This artifact changes Precision mainly as the number of DE genes increases because for smaller cutoffs, most of the genes are already significant due to the effect-based ordering. Taking this pitfall into account in our benchmark comparison, we find that a random forest using Gene Ontology features achieves the best overall results.
ER -