ProtModernBERT: Compute-Efficient Protein Language Modeling
Abstract
Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90\% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.
Related
- MI-PEFT: Mixture-of-Experts Integrated Parameter-Efficient Fine-Tuning Protein Language Models Improves Acidophilic Proteins Classification
- FedEpitope: Parameter-Efficient Federated Protein Language Models for Linear B-Cell Epitope Prediction
- Linear Fitness Subspace in Protein Language Models Enables Sample-Efficient Directed Evolution
- Joint protein-DNA language modeling enables prediction and engineering of DNA recognition specificity
- Antibody-Antigen Affinity Prediction with Chain-Aware Protein Language Modeling
BibTeX
@misc{martinez2026protmodernbert,
title = {{ProtModernBERT: Compute-Efficient Protein Language Modeling}},
author = {Alberto Martínez and Natalia Flechas Manrique and Llorenç Espinosa-Portalés and Luc Andrea},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.19738},
eprint = {2610.19738},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.19738},
}
RIS
TY - UNPB
TI - ProtModernBERT: Compute-Efficient Protein Language Modeling
AU - Alberto Martínez
AU - Natalia Flechas Manrique
AU - Llorenç Espinosa-Portalés
AU - Luc Andrea
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.19738
UR - https://arxiv.org/abs/2610.19738
AB - Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90\% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.
ER -