Re-evaluating the benefit of pretraining scale in cell foundation models
- 1Adelaide University
Abstract
Single-cell foundation models are self-supervised neural networks pretrained on millions of cellular transcriptomes to learn representations that transfer across biological domains and downstream tasks. Despite increasing investments in model and pretraining data scale, it remains unclear how to assess whether these investments improve the quality of learned biological representations. Recent evidence from standard finetuning benchmarks suggests that cell foundation models reach performance saturation after exposure to only a fraction of the available pretraining data. We hypothesised that conventional finetuning may obscure improvements in underlying representation quality and the biological information they encoded during pretraining. To address this, we tested second- and first-generation cell foundation models across corpus sizes using standard finetuning and extended it with a comprehensive fully frozen evaluation. We found that the training-data scaling relationship becomes apparent under fully frozen finetuning, particularly for second-generation models. Zero-shot analyses similarly recovered dependence on pretraining scale. Our findings reveal a fundamental limitation of conventional foundation model evaluation and establish the importance of frozen-representation benchmarking for assessing representation quality and guiding pretraining-scale decisions.
Related
- Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction
- Creator 1.0: Phylogeny-Guided Stagewise Pretraining for Plant Genome Foundation Models across 1,000 Species
- BarcodeMAE+: Rethinking Masked Pretraining and Global Representations for DNA Barcode Foundation Models
- Biobank-scale self-supervised genotype pretraining yields transferable representations for human genetics
- Technology-Assisted Rehabilitation Gym for at-home Training (TARGeT) for stroke survivors: A study protocol for evaluating feasibility, safety, usability, clinical benefit, and costs
BibTeX
@misc{shakeel2026reevaluating,
title = {{Re-evaluating the benefit of pretraining scale in cell foundation models}},
author = {M. H. Shakeel and M. Shen and S. Mangiola},
howpublished = {bioRxiv},
year = {2026},
month = jun,
doi = {10.64898/2026.06.04.730272},
url = {https://www.biorxiv.org/content/10.64898/2026.06.04.730272v2},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Re-evaluating the benefit of pretraining scale in cell foundation models
AU - M. H. Shakeel
AU - M. Shen
AU - S. Mangiola
PY - 2026
DA - 2026/06/04
PB - bioRxiv
DO - 10.64898/2026.06.04.730272
UR - https://www.biorxiv.org/content/10.64898/2026.06.04.730272v2
AB - Single-cell foundation models are self-supervised neural networks pretrained on millions of cellular transcriptomes to learn representations that transfer across biological domains and downstream tasks. Despite increasing investments in model and pretraining data scale, it remains unclear how to assess whether these investments improve the quality of learned biological representations. Recent evidence from standard finetuning benchmarks suggests that cell foundation models reach performance saturation after exposure to only a fraction of the available pretraining data. We hypothesised that conventional finetuning may obscure improvements in underlying representation quality and the biological information they encoded during pretraining. To address this, we tested second- and first-generation cell foundation models across corpus sizes using standard finetuning and extended it with a comprehensive fully frozen evaluation. We found that the training-data scaling relationship becomes apparent under fully frozen finetuning, particularly for second-generation models. Zero-shot analyses similarly recovered dependence on pretraining scale. Our findings reveal a fundamental limitation of conventional foundation model evaluation and establish the importance of frozen-representation benchmarking for assessing representation quality and guiding pretraining-scale decisions.
ER -