Scaling recipes for single-cell RNA sequencing foundation models: when do scaling laws hold?
- 1University of Oxford
Abstract
Deep learning models exhibit empirical scaling laws whereby performance changes predictably with model size, dataset size, and training compute. Although these relationships are well established in domains such as language and image modelling, their applicability to biological data remains unclear. Here, we investigate scaling behaviour in foundation models trained on large collec tions of single-cell transcriptomes. We show that pre-training loss decreases systematically with model capacity and training compute, exhibiting a power law dependence on model size. The strength and regularity of these trends differ between model formulations. We identify and quantify empirical relationships linking the optimal learning rate and depth-to-width ratio to model size and depth or compute. These results demonstrate that scaling principles extend to transcriptomic modelling. More broadly, they provide a quantitative framework for estimating the expected returns from additional resources and selecting suit able hyperparameters and architectures, thereby supporting the development of increasingly capable foundation models for omics data.
Related
- FoME: A foundation model for EEG using adaptive temporal-lateral attention scaling
- Scaling subjects in cross-modal alignment: video decoding with EEG foundation model
- Parasitism of an intertidal foundation species: Allometric scaling and environmental constraints on trematode distribution in barnacles across biological scales
- The probability of evolutionary rescue with competition: scaling-up from a single species to a community
- Scaling an Autoregressive Transformer for Single-Cell Generation
BibTeX
@misc{borra2026scaling,
title = {{Scaling recipes for single-cell RNA sequencing foundation models: when do scaling laws hold?}},
author = {F. Borra and G. Ciro' and A. Castellini and G. Gatti and A. Tangherloni and F. M. Buffa},
howpublished = {bioRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.31.747783},
url = {https://www.biorxiv.org/content/10.64898/2026.08.31.747783v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Scaling recipes for single-cell RNA sequencing foundation models: when do scaling laws hold?
AU - F. Borra
AU - G. Ciro'
AU - A. Castellini
AU - G. Gatti
AU - A. Tangherloni
AU - F. M. Buffa
PY - 2026
DA - 2026/08/31
PB - bioRxiv
DO - 10.64898/2026.08.31.747783
UR - https://www.biorxiv.org/content/10.64898/2026.08.31.747783v1
AB - Deep learning models exhibit empirical scaling laws whereby performance changes predictably with model size, dataset size, and training compute. Although these relationships are well established in domains such as language and image modelling, their applicability to biological data remains unclear. Here, we investigate scaling behaviour in foundation models trained on large collec tions of single-cell transcriptomes. We show that pre-training loss decreases systematically with model capacity and training compute, exhibiting a power law dependence on model size. The strength and regularity of these trends differ between model formulations. We identify and quantify empirical relationships linking the optimal learning rate and depth-to-width ratio to model size and depth or compute. These results demonstrate that scaling principles extend to transcriptomic modelling. More broadly, they provide a quantitative framework for estimating the expected returns from additional resources and selecting suit able hyperparameters and architectures, thereby supporting the development of increasingly capable foundation models for omics data.
ER -