Averaging in gene perturbation benchmarks hides perturbation-strength variation: Consequences for model evaluation
- 1Rutgers University
Abstract
Perturbation-response benchmarks usually represent each genetic intervention by averaging all cells assigned that perturbation label. We inquire what perturbation-strength information is lost by this averaging, and whether current model predictions preserve enough perturbation-specific information for this perturbation-strength variation to be evaluated. Using X-Atlas/Orion CRISPRi screens, we use depth-corrected guide abundance as a cell-level proxy for perturbation strength and examine how transcriptional responses vary within a nominal perturbation. Higher perturbation strength is associated with greater response reproducibility. The effect is present in two human cell lines, HCT116 and HEK293T, but is larger in HCT116. For model evaluation, HCT116 retrieval tests show that observed responses preserve perturbation identity, whereas the evaluated STATE and Cell2Sentence predictions do not reliably recover it, hence the question of within-perturbation strength information cannot be meaningfully evaluated.
Related
- Plate design confounds donor variation and model evaluation in single-cell perturbation screens
- Information contracts determine what inverse perturbation benchmarks measure
- Cause-excluded Effect Evaluation: a framework for the faithful assessment of genetic perturbation target identification
- Genetic Perturbation Prediction Models Fail Out-of-Distribution: A Rigorous Benchmark and Systematic Evaluation
- DCAF: Unifying Data Validity, Evaluation Validity and Predictive Modeling for Single-cell Perturbation Simulation
BibTeX
@misc{sharma2026averaging,
title = {{Averaging in gene perturbation benchmarks hides perturbation-strength variation: Consequences for model evaluation}},
author = {Y. S. Sharma and M. Subramaniyam and P. Passariello and D. Jay and D. Render and M. Szegedy},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.20.760537},
url = {https://www.biorxiv.org/content/10.64898/2026.10.20.760537v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Averaging in gene perturbation benchmarks hides perturbation-strength variation: Consequences for model evaluation
AU - Y. S. Sharma
AU - M. Subramaniyam
AU - P. Passariello
AU - D. Jay
AU - D. Render
AU - M. Szegedy
PY - 2026
DA - 2026/10/20
PB - bioRxiv
DO - 10.64898/2026.10.20.760537
UR - https://www.biorxiv.org/content/10.64898/2026.10.20.760537v1
AB - Perturbation-response benchmarks usually represent each genetic intervention by averaging all cells assigned that perturbation label. We inquire what perturbation-strength information is lost by this averaging, and whether current model predictions preserve enough perturbation-specific information for this perturbation-strength variation to be evaluated. Using X-Atlas/Orion CRISPRi screens, we use depth-corrected guide abundance as a cell-level proxy for perturbation strength and examine how transcriptional responses vary within a nominal perturbation. Higher perturbation strength is associated with greater response reproducibility. The effect is present in two human cell lines, HCT116 and HEK293T, but is larger in HCT116. For model evaluation, HCT116 retrieval tests show that observed responses preserve perturbation identity, whereas the evaluated STATE and Cell2Sentence predictions do not reliably recover it, hence the question of within-perturbation strength information cannot be meaningfully evaluated.
ER -