Representing biological concepts as directions in foundation model embeddings
- 1The University of Arizona
Abstract
Biological foundation models encode rich information about protein structure, function, and cellular state in high-dimensional embeddings, but accessing specific biological properties typically relies on additional, task-specific prediction models. Here, we show that diverse biological properties can be mapped to interpretable, reusable directions in embedding space. In the Concept Activation Vector (CAV) approach, examples representing a concept are distinguished from a background set using a linear model, and the resulting weight vector defines the CAV for that concept. In protein embeddings, CAVs recover Gene Ontology and Enzyme Commission annotations, localize Pfam domains, reveal novel structural relationships, and capture DNA-binding and protein-interaction interface specificity. In single-cell embeddings, CAVs isolate cancer-associated expression transitions, and gene variation along these axes substantially agrees with conventional differential-expression analysis despite being a distinct computational approach. Together, these results establish concept directions as a common representation, enabling diverse biological features to be compared, localized, and manipulated.
Related
- A Context-Aware 14-3-3 Binding Predictor Enabled by Data Augmentation, Multi-Scale Biological Features, and Protein Language Model Embeddings
- DECONVersation: Single Cell Foundation Model-Derived Embeddings for Robust Cell Type Deconvolution of Bulk RNA-seq
- Predictive dendrites as a foundation for biological learning
- BioMod: Biological Module Tokenization for Single-Cell Foundation Modeling
- Towards Trustworthy Biological Alignment in TabPFN-Probed Pathology Foundation Models
BibTeX
@misc{shamail2026representing,
title = {{Representing biological concepts as directions in foundation model embeddings}},
author = {A. Shamail and C. D. McWhite},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.10.758204},
url = {https://www.biorxiv.org/content/10.64898/2026.10.10.758204v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Representing biological concepts as directions in foundation model embeddings
AU - A. Shamail
AU - C. D. McWhite
PY - 2026
DA - 2026/10/10
PB - bioRxiv
DO - 10.64898/2026.10.10.758204
UR - https://www.biorxiv.org/content/10.64898/2026.10.10.758204v1
AB - Biological foundation models encode rich information about protein structure, function, and cellular state in high-dimensional embeddings, but accessing specific biological properties typically relies on additional, task-specific prediction models. Here, we show that diverse biological properties can be mapped to interpretable, reusable directions in embedding space. In the Concept Activation Vector (CAV) approach, examples representing a concept are distinguished from a background set using a linear model, and the resulting weight vector defines the CAV for that concept. In protein embeddings, CAVs recover Gene Ontology and Enzyme Commission annotations, localize Pfam domains, reveal novel structural relationships, and capture DNA-binding and protein-interaction interface specificity. In single-cell embeddings, CAVs isolate cancer-associated expression transitions, and gene variation along these axes substantially agrees with conventional differential-expression analysis despite being a distinct computational approach. Together, these results establish concept directions as a common representation, enabling diverse biological features to be compared, localized, and manipulated.
ER -