AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses
Abstract
Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.
Related
- Beyond the Transcriptome: Chromatin-Informed Prediction of Cell-State-Dependent Perturbation Responses
- SubcellPert: Benchmarking Subcellular Perturbation Prediction
- CellChain: Benchmarking Cross-Scale Evidence Use for Perturbation Reasoning
- Mitochondrial perturbation adapts the proteome from early to advanced immune responses
- Auditing Confidence-Calibrated Benchmarking of Single-Cell Perturbation Models
BibTeX
@misc{huang2026ambimodbench,
title = {{AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses}},
author = {Sikai Huang and Zhiwen Yang and Kai Yu and Jiayuan Chen and Stan Z. Li},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32527},
eprint = {2609.32527},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32527},
}
RIS
TY - UNPB
TI - AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses
AU - Sikai Huang
AU - Zhiwen Yang
AU - Kai Yu
AU - Jiayuan Chen
AU - Stan Z. Li
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2609.32527
UR - https://arxiv.org/abs/2609.32527
AB - Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.
ER -