UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning
- 1Texas A&M University
Abstract
Learned protein representations contribute to the broad utility of pretrained protein language models (PLMs), often enhanced by task-specific fine-tuning where parameter-efficient techniques are imperative due to large network size. Our work takes an orthogonal approach to such fine-tuning: rather than optimizing the network weights, we formulate adaptation as an optimization task over the parameterized activation space. Motivated by the prior work on curvature-tuning, we propose UPLiFT, Uncertainty Parameterized activation for Lightweight Fine-Tuning, with enhanced adaptation capacity when the underlying model--whether pretrained or randomly initialized--remains fixed, for significantly improved parameter efficiency. Furthermore, its learnable non-linearity naturally allows to model uncertainty over the activation space through fine-tuning in a variational inference framework. Evaluation across multiple protein datasets shows that UPLiFT performs competitively while requiring 3-11x fewer trainable parameters than weight-only adaptations. Bayesian UPLiFT further improves calibration while largely preserving the predictive performance of deterministically fine-tuned models. The narrow predictive performance margin with the low-rank adaptation baseline in residue-level tasks highlights the potential of our proposed approach as a parameter-efficient fine-tuning technique for PLM-based predictors.
Related
- VRPTR prediction of individual language activation and uncertainty from resting state fMRI
- The uplift of the Qinghai-Tibetan Plateau reshaped genome architecture and diversification of Asian torrent catfishes
- A lightweight privacy-aware federated vision transformer for tomato leaf disease classification
- IoT-enabled individual poultry monitoring with lightweight step-counting for health assessment
- LTK-YOLO: Lightweight Keypoint Detection for Robotic Tea Bud Picking
BibTeX
@misc{abeer2026uplift,
title = {{UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning}},
author = {A. N. M. N. Abeer and A. H. Rahmati and X. Qian and B.-J. Yoon},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.05.756747},
url = {https://www.biorxiv.org/content/10.64898/2026.10.05.756747v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning
AU - A. N. M. N. Abeer
AU - A. H. Rahmati
AU - X. Qian
AU - B.-J. Yoon
PY - 2026
DA - 2026/10/05
PB - bioRxiv
DO - 10.64898/2026.10.05.756747
UR - https://www.biorxiv.org/content/10.64898/2026.10.05.756747v1
AB - Learned protein representations contribute to the broad utility of pretrained protein language models (PLMs), often enhanced by task-specific fine-tuning where parameter-efficient techniques are imperative due to large network size. Our work takes an orthogonal approach to such fine-tuning: rather than optimizing the network weights, we formulate adaptation as an optimization task over the parameterized activation space. Motivated by the prior work on curvature-tuning, we propose UPLiFT, Uncertainty Parameterized activation for Lightweight Fine-Tuning, with enhanced adaptation capacity when the underlying model--whether pretrained or randomly initialized--remains fixed, for significantly improved parameter efficiency. Furthermore, its learnable non-linearity naturally allows to model uncertainty over the activation space through fine-tuning in a variational inference framework. Evaluation across multiple protein datasets shows that UPLiFT performs competitively while requiring 3-11x fewer trainable parameters than weight-only adaptations. Bayesian UPLiFT further improves calibration while largely preserving the predictive performance of deterministically fine-tuned models. The narrow predictive performance margin with the low-rank adaptation baseline in residue-level tasks highlights the potential of our proposed approach as a parameter-efficient fine-tuning technique for PLM-based predictors.
ER -