Cell-level random splits leak group-owned answers in single-cell benchmarks
- 1None
Abstract
Machine learning models in single-cell biology increasingly forecast differentiation, reprogramming and therapeutic response from early transcriptomic profiles. Testing whether a model has learned real biology requires held-out cells. Single-cell data, however, are grouped: cells from the same clone, patient or batch share the same label. A random split therefore places relatives of each test cell, carrying its label, in the training set, and a model can score well by memorizing a relative instead of learning a transferable rule. Grouped validation removes this leakage but leaves far fewer independent units behind each error bar. Here we show how to estimate this leakage before training any model, from two properties of the data: exposure, the fraction of test cells with relatives in training, and retrievability, how often a nearest-neighbor search returns such a relative rather than an unrelated cell. Across lineage-barcoded and patient data, exposure determines whether a random split opens a leakage channel, and retrievability determines how much it can inflate the score. The inflation is negligible where cell state has decoupled from ancestry, much larger where clonal sisters remain close in expression space, and in a patient cohort large enough to overturn a clinical conclusion. We also provide leakcheck, which computes both properties in seconds, before the outcome model is fitted.
Related
- MutaStruct: Revisiting Benchmarks for Single-Site Mutation-Induced Structural Change Prediction
- A lifespan single-cell atlas of the human developing hippocampus benchmarks familial Alzheimer's disease brain organoids.
- Evaluating the Robustness of Path-Preservation Benchmarks for Dimensionality Reduction Across Point-Density Thresholds: Linear and Cyclic Single-Cell Trajectories
- Predicted and experimental protein-ligand coordinates with distance labels and evaluation splits
- Beyond Scaffold Splits: Structural-Frontier Evaluation Reveals Hidden Failures in ADMET Models
BibTeX
@misc{sun2026celllevel,
title = {{Cell-level random splits leak group-owned answers in single-cell benchmarks}},
author = {S. Sun and H. Cang},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.09.750484},
url = {https://www.biorxiv.org/content/10.64898/2026.09.09.750484v1},
note = {Public domain},
}
RIS
TY - UNPB
TI - Cell-level random splits leak group-owned answers in single-cell benchmarks
AU - S. Sun
AU - H. Cang
PY - 2026
DA - 2026/09/09
PB - bioRxiv
DO - 10.64898/2026.09.09.750484
UR - https://www.biorxiv.org/content/10.64898/2026.09.09.750484v1
AB - Machine learning models in single-cell biology increasingly forecast differentiation, reprogramming and therapeutic response from early transcriptomic profiles. Testing whether a model has learned real biology requires held-out cells. Single-cell data, however, are grouped: cells from the same clone, patient or batch share the same label. A random split therefore places relatives of each test cell, carrying its label, in the training set, and a model can score well by memorizing a relative instead of learning a transferable rule. Grouped validation removes this leakage but leaves far fewer independent units behind each error bar. Here we show how to estimate this leakage before training any model, from two properties of the data: exposure, the fraction of test cells with relatives in training, and retrievability, how often a nearest-neighbor search returns such a relative rather than an unrelated cell. Across lineage-barcoded and patient data, exposure determines whether a random split opens a leakage channel, and retrievability determines how much it can inflate the score. The inflation is negligible where cell state has decoupled from ancestry, much larger where clonal sisters remain close in expression space, and in a patient cohort large enough to overturn a clinical conclusion. We also provide leakcheck, which computes both properties in seconds, before the outcome model is fitted.
ER -