Scaling an Autoregressive Transformer for Single-Cell Generation
Abstract
We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.
Related
- Scaling recipes for single-cell RNA sequencing foundation models: when do scaling laws hold?
- De novo transformer modeling improves recovery of genetic cell types from sparse single-cell RNA sequencing
- Benchmarking Recent Single-Cell Synthetic Data Generation Methods on the OneK1K Dataset
- Integrative reanalysis of public intracellular and vaccine-challenge transcriptomes, informed by single-cell and metabolomic atlases, refines antigen prioritisation for next-generation veterinary Brucella vaccines
- RAGCell: Retrieval-Augmented Generation as Supervision for Versatile Single-cell Analysis
BibTeX
@misc{sharipov2026scaling,
title = {{Scaling an Autoregressive Transformer for Single-Cell Generation}},
author = {Aleksandr Sharipov and Yusif Mukhtarov and Igor Molybog},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.02961},
eprint = {2608.02961},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.02961},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Scaling an Autoregressive Transformer for Single-Cell Generation
AU - Aleksandr Sharipov
AU - Yusif Mukhtarov
AU - Igor Molybog
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2608.02961
UR - https://arxiv.org/abs/2608.02961
AB - We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.
ER -