scArchon: a scalable benchmarking framework for assessing single-cell perturbation models
6 affiliations
- 1BioQuant, Heidelberg University, Heidelberg, Germany. [email protected]
- 2Institute of Pharmacy and Molecular Biotechnology (IPMB), Heidelberg University, Heidelberg, Germany. [email protected]
- 3BioQuant, Heidelberg University, Heidelberg, Germany
- 4Institute of Pharmacy and Molecular Biotechnology (IPMB), Heidelberg University, Heidelberg, Germany
- 5BioQuant, Heidelberg University, Heidelberg, Germany. [email protected]
- 6Institute of Pharmacy and Molecular Biotechnology (IPMB), Heidelberg University, Heidelberg, Germany. [email protected]
Abstract
BACKGROUND: The accurate prediction of cellular responses to perturbations, such as drug treatments, remains a pivotal challenge in single-cell transcriptomics. While numerous deep learning tools have been developed for this task, their systematic benchmarking across diverse datasets and performance metrics has been limited. RESULTS: Here, we present scArchon, a reproducible, modular benchmarking platform built on Snakemake. It is designed to evaluate perturbation response prediction tools in an unbiased and extensible manner. Employing six representative single-cell RNA-seq datasets, we compare leading methods such as scGen, CPA, trVAE, scPRAM, scVIDR, scDisInFact, SCREEN, scPreGAN, and CellOT against baselines. We assess model performance using a composite of statistical and biological metrics. Our analysis reveals heterogeneous performance. While methods like trVAE, scGen, scPRAM, and scVIDR achieve robust results across multiple datasets, other tools occasionally underperform even compared to linear or control baselines. Notably, models with favorable quantitative scores may fail to retain key biological perturbation signatures, underscoring the need for gene-level evaluation. CONCLUSIONS: scArchon provides a unified, extensible foundation for large-scale, standardized benchmarking of perturbation prediction tools, facilitating methodological transparency and accelerating development in this rapidly evolving field. We encourage adoption of scArchon and sharing of containerized tools to drive progress in single-cell perturbation modeling.
Related
- Dynamic perturbation analysis: a modular rewiring framework for decoding cancer stage progression and identifying therapeutic vulnerabilities.
- VHH-Bench: A Benchmarking Framework for Evaluating Nanobody–Antigen Complex Predictions Across CDR3 Conformations
- Benchmarking batch-effect correction in clinical bead-based multiplex immunoassays: a framework and web tool for cytokine and soluble immune checkpoint data
- Quantitative lipidomics reveals Lipid perturbation of plasma apoptotic extracellular vesicles during physiological aging
- Localized thermal perturbation alters lipid bilayer structure and daunorubicin interactions in MscL-containing membrane systems
BibTeX
@article{radig2026scarchon,
title = {{scArchon: a scalable benchmarking framework for assessing single-cell perturbation models}},
author = {Jean Radig and Robin Droit and Daria Doncevic and Albert Li and Duc Thien Bui and Luis Herfurth and Thaddeus Kühn and Carl Herrmann},
journal = {Genome biology},
year = {2026},
month = may,
doi = {10.1186/s13059-026-04104-z},
url = {https://doi.org/10.1186/s13059-026-04104-z},
note = {CC BY},
}
RIS
TY - JOUR
TI - scArchon: a scalable benchmarking framework for assessing single-cell perturbation models
AU - Jean Radig
AU - Robin Droit
AU - Daria Doncevic
AU - Albert Li
AU - Duc Thien Bui
AU - Luis Herfurth
AU - Thaddeus Kühn
AU - Carl Herrmann
PY - 2026
DA - 2026/05/12
JO - Genome biology
DO - 10.1186/s13059-026-04104-z
UR - https://doi.org/10.1186/s13059-026-04104-z
AB - BACKGROUND: The accurate prediction of cellular responses to perturbations, such as drug treatments, remains a pivotal challenge in single-cell transcriptomics. While numerous deep learning tools have been developed for this task, their systematic benchmarking across diverse datasets and performance metrics has been limited. RESULTS: Here, we present scArchon, a reproducible, modular benchmarking platform built on Snakemake. It is designed to evaluate perturbation response prediction tools in an unbiased and extensible manner. Employing six representative single-cell RNA-seq datasets, we compare leading methods such as scGen, CPA, trVAE, scPRAM, scVIDR, scDisInFact, SCREEN, scPreGAN, and CellOT against baselines. We assess model performance using a composite of statistical and biological metrics. Our analysis reveals heterogeneous performance. While methods like trVAE, scGen, scPRAM, and scVIDR achieve robust results across multiple datasets, other tools occasionally underperform even compared to linear or control baselines. Notably, models with favorable quantitative scores may fail to retain key biological perturbation signatures, underscoring the need for gene-level evaluation. CONCLUSIONS: scArchon provides a unified, extensible foundation for large-scale, standardized benchmarking of perturbation prediction tools, facilitating methodological transparency and accelerating development in this rapidly evolving field. We encourage adoption of scArchon and sharing of containerized tools to drive progress in single-cell perturbation modeling.
SN - 1465-6906
SN - 1465-6914
SN - 1474-7596
SN - 1474-760X
ER -