Cell-type separability predicts annotation accuracy and outweighs algorithm choice: a factorial benchmark across seven scRNA paradigms
- 1University of Notre Dame
Abstract
Automated cell-type annotation is a prerequisite for nearly all single-cell RNA-sequencing (scRNA-seq) analysis, and the proliferation of annotation tools -- spanning marker-based, similarity-based, classical machine-learning, deep-learning, semi-supervised, large language model (LLM), and transformer foundation-model families -- has exceeded any existing guidance on how to select among them. Existing benchmarks utilize convenient, well-known datasets in which cell number, class imbalance, cell-type number, and differential-expression strength vary concurrently, so performance cannot be attributed to any dataset property. We benchmarked 63 tools from seven families using a Taguchi L9(34) orthogonal array that varied these four properties independently and extended the comparison under more realistic conditions: real data, cross-platform transfer, public marker databases, LLM annotation, and fine-tuned foundation models. With standardized preprocessing and inputs, the leading cell-level families performed within 0.06 {kappa} of one another, and fine-tuned foundation models averaged only slightly higher. Across families, sequencing platforms, and data types, accuracy was predicted near-linearly by how separable cell types were in a shared expression embedding space, measured as k-nearest-neighbor (kNN) purity (R2 = 0.85-0.99). We found that dataset identity accounted for {approx}84% of {kappa} variance while tool identity accounted for {approx}4%. Strong performers were distributed across families, for example, Seurat label transfer and singleCellNet among reference-based classifiers and mLLMCelltype among LLM annotators. When cell types cleanly separate, many annotation methods suffice; when they do not, algorithmic complexity struggles to compensate for noisy signal. Method selection should therefore prioritize available resources, such as curated references and compute availability, over sophisticated modeling.
Related
- Where single-cell foundation models help: a benchmark of annotation, perturbation and regulatory inference against simple baselines
- An annotation-overlap-flagged rare-disease gene-prioritisation benchmark and PMC index recipe
- MillionTrees: A global benchmark for tree detection across geography, sensor, and annotation geometry
- Hydrogeological compartmentalization outweighs hydrochemistry in structuring microbial communities across contrasting subsurface habitats
- Landscape configuration outweighs composition in shaping the diversity of phytophagous scarab beetles in the Serranía del Perijá, Colombia
BibTeX
@misc{wardhana2026celltype,
title = {{Cell-type separability predicts annotation accuracy and outweighs algorithm choice: a factorial benchmark across seven scRNA paradigms}},
author = {O. Wardhana and Z. Zeng and X. Lu},
howpublished = {bioRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.28.747622},
url = {https://www.biorxiv.org/content/10.64898/2026.08.28.747622v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Cell-type separability predicts annotation accuracy and outweighs algorithm choice: a factorial benchmark across seven scRNA paradigms
AU - O. Wardhana
AU - Z. Zeng
AU - X. Lu
PY - 2026
DA - 2026/08/28
PB - bioRxiv
DO - 10.64898/2026.08.28.747622
UR - https://www.biorxiv.org/content/10.64898/2026.08.28.747622v1
AB - Automated cell-type annotation is a prerequisite for nearly all single-cell RNA-sequencing (scRNA-seq) analysis, and the proliferation of annotation tools -- spanning marker-based, similarity-based, classical machine-learning, deep-learning, semi-supervised, large language model (LLM), and transformer foundation-model families -- has exceeded any existing guidance on how to select among them. Existing benchmarks utilize convenient, well-known datasets in which cell number, class imbalance, cell-type number, and differential-expression strength vary concurrently, so performance cannot be attributed to any dataset property. We benchmarked 63 tools from seven families using a Taguchi L9(34) orthogonal array that varied these four properties independently and extended the comparison under more realistic conditions: real data, cross-platform transfer, public marker databases, LLM annotation, and fine-tuned foundation models. With standardized preprocessing and inputs, the leading cell-level families performed within 0.06 {kappa} of one another, and fine-tuned foundation models averaged only slightly higher. Across families, sequencing platforms, and data types, accuracy was predicted near-linearly by how separable cell types were in a shared expression embedding space, measured as k-nearest-neighbor (kNN) purity (R2 = 0.85-0.99). We found that dataset identity accounted for {approx}84% of {kappa} variance while tool identity accounted for {approx}4%. Strong performers were distributed across families, for example, Seurat label transfer and singleCellNet among reference-based classifiers and mLLMCelltype among LLM annotators. When cell types cleanly separate, many annotation methods suffice; when they do not, algorithmic complexity struggles to compensate for noisy signal. Method selection should therefore prioritize available resources, such as curated references and compute availability, over sophisticated modeling.
ER -