Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation
Abstract
Single-cell and spatial foundation models promise transferable biological representations, yet their generality remains largely untested across modalities, biological domains and analytical tasks. We benchmarked six representative models, Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo and Novae, using a harmonised framework spanning scRNA-seq, spatial transcriptomics and Perturb-seq. We evaluated zero-shot and continually pretrained clustering, supervised annotation, marker-gene concordance and perturbation prediction. Model performance was strongly conditional: expression-trained cell-level transformers best resolved many cell-identity tasks, spatial and graph-aware models better preserved tissue architecture, and language-derived gene embeddings were competitive for selected perturbation-response metrics. No model dominated across tasks, and rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice. This benchmark provides practical guidance for model selection and argues that future models should be judged by biological generalisation, interpretability and perturbation-grounded validity, not by scale or leaderboard performance alone.
Related
- Do sEMG Foundation Models Improve Cross-Subject Generalisation?
- OncoSpot enables benchmarking of cancer spatial transcriptomics methods across tumor evolutionary modes
- Decision benchmarking reveals when protein foundation models add experimental value
- Benchmarking single-cell foundation models reveals an advantage in the low-data regime for cell-type and disease-state prediction
- Systematic benchmarking of commercial workflows for isoform-resolved single-nucleus transcriptomics
BibTeX
@misc{chen2026harmonised,
title = {{Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation}},
author = {Sally Chen and Roxana Zahedi and Lucy Chhuo and Ricky Nguyen and Marjan BaghGolshani and Amin Beheshti and Mark Grosser and Min Yang and Nona Farbehi and Nigel Lovell and Ahmadreza Argha and Thantrira Porntaveetusm and Fatemeh Vafaee and Youqiong Ye and Hamid Alinejad-Rokny},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.17227},
eprint = {2607.17227},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.17227},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation
AU - Sally Chen
AU - Roxana Zahedi
AU - Lucy Chhuo
AU - Ricky Nguyen
AU - Marjan BaghGolshani
AU - Amin Beheshti
AU - Mark Grosser
AU - Min Yang
AU - Nona Farbehi
AU - Nigel Lovell
AU - Ahmadreza Argha
AU - Thantrira Porntaveetusm
AU - Fatemeh Vafaee
AU - Youqiong Ye
AU - Hamid Alinejad-Rokny
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2607.17227
UR - https://arxiv.org/abs/2607.17227
AB - Single-cell and spatial foundation models promise transferable biological representations, yet their generality remains largely untested across modalities, biological domains and analytical tasks. We benchmarked six representative models, Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo and Novae, using a harmonised framework spanning scRNA-seq, spatial transcriptomics and Perturb-seq. We evaluated zero-shot and continually pretrained clustering, supervised annotation, marker-gene concordance and perturbation prediction. Model performance was strongly conditional: expression-trained cell-level transformers best resolved many cell-identity tasks, spatial and graph-aware models better preserved tissue architecture, and language-derived gene embeddings were competitive for selected perturbation-response metrics. No model dominated across tasks, and rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice. This benchmark provides practical guidance for model selection and argues that future models should be judged by biological generalisation, interpretability and perturbation-grounded validity, not by scale or leaderboard performance alone.
ER -