Evaluation design shapes estimates of cross-donor generalization in single-cell classification of cell-type
Abstract
Motivation: Single-cell cell-type classifiers are often evaluated using random cell-level splits, although cells from the same donor are not independent. We asked how evaluation design affects estimates of cross-donor generalization. Results: Across PBMC and lung datasets, random cell-level splitting consistently yielded higher macro F1 than donor-held-out evaluation across multiple expression representations. This gap persisted after batch covariate adjustment and donor-level mixed-effects modeling. In contrast, evaluation gaps were minimal in near-ceiling kidney, pancreas, and blood benchmarks and in a randomized-label floor control. These results show that evaluation bias depends on the underlying classification regime and that donor-held-out evaluation is necessary when claims concern generalization to unseen individuals. Availability and Implementation: Code and analyses are available on GitHub at https://github.com/danielchen05/scRNA-cross-donor-generalization/ and archived at Zenodo at 10.5281/zenodo.23669184. All data are publicly available and cited.
Related
- How Meaning Shapes a Rule: Neural Signatures of Morphological Learning and Generalization
- Scan-sampling interval shapes the reliability of immersion-activity estimates in captive penguins
- Decision Confidence Neuron in Echo State Network for Continual Evaluation of EEG Motor Imagery Classification Quality
- De Novo Design and AlphaFold3 Evaluation of Protein Binders Targeting Specific Sites of MAP4K4
- Comprehensive evaluation of antibody language models across multiple antibody design tasks
BibTeX
@misc{chen2026evaluation,
title = {{Evaluation design shapes estimates of cross-donor generalization in single-cell classification of cell-type}},
author = {Xingyi Chen and Jiabei Li and Alexis Battle},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.26.761949},
url = {https://www.biorxiv.org/content/10.64898/2026.10.26.761949v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Evaluation design shapes estimates of cross-donor generalization in single-cell classification of cell-type
AU - Xingyi Chen
AU - Jiabei Li
AU - Alexis Battle
PY - 2026
DA - 2026/10/26
PB - bioRxiv
DO - 10.64898/2026.10.26.761949
UR - https://www.biorxiv.org/content/10.64898/2026.10.26.761949v1
AB - Motivation: Single-cell cell-type classifiers are often evaluated using random cell-level splits, although cells from the same donor are not independent. We asked how evaluation design affects estimates of cross-donor generalization. Results: Across PBMC and lung datasets, random cell-level splitting consistently yielded higher macro F1 than donor-held-out evaluation across multiple expression representations. This gap persisted after batch covariate adjustment and donor-level mixed-effects modeling. In contrast, evaluation gaps were minimal in near-ceiling kidney, pancreas, and blood benchmarks and in a randomized-label floor control. These results show that evaluation bias depends on the underlying classification regime and that donor-held-out evaluation is necessary when claims concern generalization to unseen individuals. Availability and Implementation: Code and analyses are available on GitHub at https://github.com/danielchen05/scRNA-cross-donor-generalization/ and archived at Zenodo at 10.5281/zenodo.23669184. All data are publicly available and cited.
SN - 2692-8205
ER -