Task- and dataset-specific information in protein language models
Abstract
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.
Related
- Efficient structural information integration into protein language models with implicit mixture-of-experts ProBEiT
- Learning Task-Specific Antibody Representations via Function-Aware Masking
- CAFNet-DG: Task-Specific Modeling and Prevalence-Aware Evaluation for Adverse Drug Reaction Prioritization with Ordinal-Frequency Estimation.
- Dataset-specific retention time adjustment and multi-evidence lipid annotation: a reproducible R workflow for untargeted LC–MS lipidomics
- Understanding language model scaling for protein fitness prediction
BibTeX
@misc{joeres2026task,
title = {{Task- and dataset-specific information in protein language models}},
author = {Roman Joeres and Ilya Senatorov and Anastasia Kolchina and Dietrich Klakow and Olga V. Kalinina},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.12090},
eprint = {2608.12090},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.12090},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Task- and dataset-specific information in protein language models
AU - Roman Joeres
AU - Ilya Senatorov
AU - Anastasia Kolchina
AU - Dietrich Klakow
AU - Olga V. Kalinina
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2608.12090
UR - https://arxiv.org/abs/2608.12090
AB - Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.
ER -