CellMSA: Context Modeling for Single-Cell Representation Learning
Abstract
Single-cell transcriptomics enables profiling of cellular states at unprecedented resolution, but its high dimensionality, sparsity, and technical batch effects pose significant challenges for representation learning. Existing single-cell foundation models typically encode each cell independently or only model cells from the same batch for denoising, thereby underutilizing the rich relational information across batches and cell types to model gene expression patterns. We argue that single-cell models can benefit from more informative cell-context modeling. By comparing consistency and variation across cells, models can capture fine-grained gene-gene dependencies associated with cell states, which are essential for learning high-quality representations. Inspired by the use of multiple sequence alignment (MSA) context in protein modeling, we propose CellMSA, a single-cell representation learning framework that introduces an MSA-inspired inductive bias into transcriptomic modeling. For each target cell, CellMSA retrieves relevant cells from different batches and biologically related cell types as context, and summarizes cross-cell patterns into a context-dependent gene-pair representation. This representation is then injected into a pair-aware target-cell encoder for fine-grained representation learning. We pretrain CellMSA on a large-scale human single-cell corpus of approximately 109 million cell observations, including 65.6 million primary observations. Experiments show that our framework consistently outperforms existing methods across multiple benchmarks. Code is available at the following repository: https://github.com/PharMolix/CellMSA.
Related
- A context-aware temporal modeling through unified multi-scale temporal encoding and hierarchical sequence learning for single-channel EEG sleep staging
- Concept-aware contrastive representation learning for cross-translation semantic consistency modeling of the Analects
- Extracting interpretable single-cell metabolic states with graph-guided representation learning
- Single-channel EEG sleep stage classification using synchrosqueezed transform and sequential representation learning
- scRep: A Latent-Space Self-Distilled Foundation Model for Single-Cell Representation Learning
BibTeX
@misc{zhao2026cellmsa,
title = {{CellMSA: Context Modeling for Single-Cell Representation Learning}},
author = {Suyuan Zhao and Minghao Liu and Yizhen Luo and Zaiqing Nie},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38908},
eprint = {2609.38908},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38908},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - CellMSA: Context Modeling for Single-Cell Representation Learning
AU - Suyuan Zhao
AU - Minghao Liu
AU - Yizhen Luo
AU - Zaiqing Nie
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.38908
UR - https://arxiv.org/abs/2609.38908
AB - Single-cell transcriptomics enables profiling of cellular states at unprecedented resolution, but its high dimensionality, sparsity, and technical batch effects pose significant challenges for representation learning. Existing single-cell foundation models typically encode each cell independently or only model cells from the same batch for denoising, thereby underutilizing the rich relational information across batches and cell types to model gene expression patterns. We argue that single-cell models can benefit from more informative cell-context modeling. By comparing consistency and variation across cells, models can capture fine-grained gene-gene dependencies associated with cell states, which are essential for learning high-quality representations. Inspired by the use of multiple sequence alignment (MSA) context in protein modeling, we propose CellMSA, a single-cell representation learning framework that introduces an MSA-inspired inductive bias into transcriptomic modeling. For each target cell, CellMSA retrieves relevant cells from different batches and biologically related cell types as context, and summarizes cross-cell patterns into a context-dependent gene-pair representation. This representation is then injected into a pair-aware target-cell encoder for fine-grained representation learning. We pretrain CellMSA on a large-scale human single-cell corpus of approximately 109 million cell observations, including 65.6 million primary observations. Experiments show that our framework consistently outperforms existing methods across multiple benchmarks. Code is available at the following repository: https://github.com/PharMolix/CellMSA.
ER -