Analysis of Quantized and Efficiently Adapted Protein Language Models

Abstract

Background: Protein language models (PLMs) are increasingly used for sequence generation and property prediction, but their size makes fine-tuning and deployment expensive. The effects of quantization and parameter efficient fine-tuning on performance, representations and generation remain insufficiently characterized. Results: We evaluated 4-bit quantization and low-rank adapter fine-tuning (QLoRA) across ESM-2, ESMC, ProtBERT, ProtT5, Ankh, Ankh3 and Profluent-E1. Across protein prediction tasks, many model-task pairs retained more than 90% of full fine-tuning performance. Peak GPU memory savings approached 90% for the largest models, although performance and efficiency varied by model, dataset and training configuration. QLoRA often preserved early-layer representations while inducing task-specific adaptations in middle and late layers, resembling full fine-tuning with smaller representational changes. Training speed and power effects were more varied. For unconditional generation with ProLLaMA, ProtGPT2, ProGen2, ProteinGLM and ESM3, 4-bit quantization largely preserved predicted structural and sequence-level properties, but token-level analysis revealed model-dependent shifts in autoregressive output distributions. Conclusion: QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage. Our results support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery. For generative PLMs, sequence-level and structural metrics should be complemented with distributional analysis, since downstream predictions alone may miss quantization-induced shifts. These approaches can broaden access to large-scale protein modelling while requiring model- and task-specific validation.

BananaVLM: A Domain-Adapted Vision Language Model for Banana Crop Disease Diagnosis
Sangam Kumar Jena, Pandarasamy Arjunan
arXiv1 Sep 2026PreprintCC BY-SA 4.0Full textBioinformatics & computational biologyarXiv
Integrating large language model-generated culturally adapted verbal suggestions into music therapy for university students: A pilot randomized exploratory study
Wen Li, Guoqing Wang, Pravina Manoharan +3
PLOS One9 Oct 2026Peer-reviewedCC BY 4.0Full textNeuroscience Cardiovascular medicinePLOS
Limitations of general protein language models and public TCRpMHC data in training specificity prediction models
B. Rai, S. Hall-Swan
bioRxiv21 Sep 2026PreprintAll rights reservedFull textBioinformatics & computational biologybioRxiv
Protein Language Model-Conditioned Graph Neural Networks for Multitask GPCR Ligand Activity Prediction
M. De, E. Lodh, S. Majumder +1
bioRxiv27 Sep 2026PreprintAll rights reservedFull textBioinformatics & computational biologybioRxiv
A Context-Aware 14-3-3 Binding Predictor Enabled by Data Augmentation, Multi-Scale Biological Features, and Protein Language Model Embeddings
L. Liu, Z. Wang, X. Huang
bioRxiv18 Sep 2026PreprintAll rights reservedFull textBioinformatics & computational biologybioRxiv

Cite

BibTeX RIS

BibTeX

@misc{zeisler2026analysis,
  title = {{Analysis of Quantized and Efficiently Adapted Protein Language Models}},
  author = {Ilan Yaniv Zeisler and Sebastian Clancy and Pouriya Bayat and Saaim Raad and Ivan Kraskov and Matthew Xie and Vivian White and Spencer Perkins and Serena Singh and Sepehr Bayat and Keith Pardee},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.00665},
  eprint = {2610.00665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.00665},
}

RIS

TY  - UNPB
TI  - Analysis of Quantized and Efficiently Adapted Protein Language Models
AU  - Ilan Yaniv Zeisler
AU  - Sebastian Clancy
AU  - Pouriya Bayat
AU  - Saaim Raad
AU  - Ivan Kraskov
AU  - Matthew Xie
AU  - Vivian White
AU  - Spencer Perkins
AU  - Serena Singh
AU  - Sepehr Bayat
AU  - Keith Pardee
PY  - 2026
DA  - 2026/10/01
PB  - arXiv
DO  - 10.48550/arxiv.2610.00665
UR  - https://arxiv.org/abs/2610.00665
AB  - Background: Protein language models (PLMs) are increasingly used for sequence generation and property prediction, but their size makes fine-tuning and deployment expensive. The effects of quantization and parameter efficient fine-tuning on performance, representations and generation remain insufficiently characterized. Results: We evaluated 4-bit quantization and low-rank adapter fine-tuning (QLoRA) across ESM-2, ESMC, ProtBERT, ProtT5, Ankh, Ankh3 and Profluent-E1. Across protein prediction tasks, many model-task pairs retained more than 90% of full fine-tuning performance. Peak GPU memory savings approached 90% for the largest models, although performance and efficiency varied by model, dataset and training configuration. QLoRA often preserved early-layer representations while inducing task-specific adaptations in middle and late layers, resembling full fine-tuning with smaller representational changes. Training speed and power effects were more varied. For unconditional generation with ProLLaMA, ProtGPT2, ProGen2, ProteinGLM and ESM3, 4-bit quantization largely preserved predicted structural and sequence-level properties, but token-level analysis revealed model-dependent shifts in autoregressive output distributions. Conclusion: QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage. Our results support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery. For generative PLMs, sequence-level and structural metrics should be complemented with distributional analysis, since downstream predictions alone may miss quantization-induced shifts. These approaches can broaden access to large-scale protein modelling while requiring model- and task-specific validation.
ER  -

Abstract

Related

Cite