Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction
- 1Department of Computational Medicine and Bioinformatics, University of Michigan
Abstract
Single-cell foundation models (scFMs) are commonly benchmarked using final-layer embeddings, although useful depth may depend on the task. We evaluate Geneformer, scBERT, scGPT, and scFoundation as frozen feature extractors for cell-type and perturbation-response prediction using layer-wise probes, leave-one-dataset-out (LODO) selection, readout controls, representation diagnostics, centered kernel alignment (CKA), and retrieval/collapse analyses. Intermediate layers significantly improve cell-type accuracy for scGPT and scBERT in several settings, while Geneformer and scFoundation generally show performance comparable to their final layers. For perturbation prediction, LODO identifies layer 1 as an earliest-on-par candidate across all four models, without establishing universal superiority. Paired scGPT CLS and mean-pooling analyses show that readout affects preferred depth. CKA and representation metrics reveal model- and task-dependent geometric changes, while retrieval and collapse checks support the robustness of perturbation findings. We recommend reporting extraction depth, readout, selection protocol, and task-specific diagnostics. A validated non-final layer may also reduce computation, although efficiency was not directly benchmarked.
Related
- Evaluating Autonomous LLM Agents Across Molecular Prediction and Optimization Benchmarks
- Re-evaluating the benefit of pretraining scale in cell foundation models
- MtReNIRS: A Self-Supervised Foundation Model for Transferable Neurovascular Representations
- BarcodeMAE+: Rethinking Masked Pretraining and Global Representations for DNA Barcode Foundation Models
- Beyond compliance: evaluating the impact of assent-based training on equine behavior and the human-horse relationship.
BibTeX
@misc{wen2026beyond,
title = {{Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction}},
author = {M.-C. C. Wen and Y. Lyu and A. Robbins},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.05.756880},
url = {https://www.biorxiv.org/content/10.64898/2026.10.05.756880v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction
AU - M.-C. C. Wen
AU - Y. Lyu
AU - A. Robbins
PY - 2026
DA - 2026/10/05
PB - bioRxiv
DO - 10.64898/2026.10.05.756880
UR - https://www.biorxiv.org/content/10.64898/2026.10.05.756880v1
AB - Single-cell foundation models (scFMs) are commonly benchmarked using final-layer embeddings, although useful depth may depend on the task. We evaluate Geneformer, scBERT, scGPT, and scFoundation as frozen feature extractors for cell-type and perturbation-response prediction using layer-wise probes, leave-one-dataset-out (LODO) selection, readout controls, representation diagnostics, centered kernel alignment (CKA), and retrieval/collapse analyses. Intermediate layers significantly improve cell-type accuracy for scGPT and scBERT in several settings, while Geneformer and scFoundation generally show performance comparable to their final layers. For perturbation prediction, LODO identifies layer 1 as an earliest-on-par candidate across all four models, without establishing universal superiority. Paired scGPT CLS and mean-pooling analyses show that readout affects preferred depth. CKA and representation metrics reveal model- and task-dependent geometric changes, while retrieval and collapse checks support the robustness of perturbation findings. We recommend reporting extraction depth, readout, selection protocol, and task-specific diagnostics. A validated non-final layer may also reduce computation, although efficiency was not directly benchmarked.
ER -