PFArena: Benchmarking Language Models for Protein Modification

Abstract

Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational paradigms including protein language models (PLMs), large language models (LLMs), and LLM-based agents have shown promise in protein modification, their relative efficacy across realistic experimental decision-making settings remains unclear. To bridge this gap, we introduce PFArena, a benchmark comprising four controlled task interfaces that cover single-mutant generation and multi-mutant ranking. By providing varying levels of mutation fitness data, PFArena reflects four representative research scenarios characterized by differing degrees of prior experimental context. We assess six PLMs, six LLMs, and five LLM-based agents using complementary metrics to measure both peak and overall protein modification performance. Our evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data are available. Nevertheless, all model families face fundamental challenges with increasing search-space size and mutation depth. We release our code and benchmark suite to facilitate reproducible research in model-assisted protein modification.

Modification of porcine plasma protein structure and flavor by ultrasound-assisted enzymatic hydrolysis.
Lingrui Li, Jingwei Pan, Tiantian Wu +4
Ultrasonics sonochemistry12 Oct 2026Peer-reviewedCC BYFull textMolecular biology & gene regulationEurope PMC
Benzimidazole Modification of DNA Aptamers for Ultra Responsive Protein-Induced Fluorescence Enhancement
Matteo Pavlovic, Austin Pounder, Naomi Varnovitski +2
ChemRxiv15 Sep 2026PreprintCC BYFull textMolecular biology & gene regulationChemRxiv
Static Recovery Is Not Dynamic Stability: Dynamics-Aware Benchmarking of Protein Motif Scaffolding
Emile de Bruyn, Moritz Schäffler, Jannik Schneider +9
arXiv5 Nov 2026PreprintarXiv licenceFull textBiophysicsarXiv
Understanding language model scaling for protein fitness prediction
Chao Hou, Ди Лю, Aziz Zafar +1
Nature Computational Science13 Jul 2026Peer-reviewedLicence not statedFull text19 citedMolecular biology & gene regulationOpenAlex
Interlaboratory benchmarking of differential scanning fluorimetry reveals sources of variability in protein thermal stability parameters
S. Brule, M. Chevreuil, O. Burastero +40
bioRxiv12 Oct 2026PreprintCC BYFull textBiophysicsbioRxiv

Cite

BibTeX RIS

BibTeX

@misc{ouyang2026pfarena,
  title = {{PFArena: Benchmarking Language Models for Protein Modification}},
  author = {Yawen Ouyang and Xinbo Zhang and Ziyuan Ma and Yixin Wu and Wenbin Liao and Feiran Zhang and Wenjie Li and Lihao Wang and Hao Wang and Xiaoqing Zheng and Xuefeng Yan and Lei Bai and Ya-Qin Zhang and Shuyi Zhang and Wei-Ying Ma and Dahua Lin and Bowen Zhou and Hao Zhou},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.28921},
  eprint = {2609.28921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.28921},
  note = {CC BY-NC-SA 4.0},
}

RIS

TY  - UNPB
TI  - PFArena: Benchmarking Language Models for Protein Modification
AU  - Yawen Ouyang
AU  - Xinbo Zhang
AU  - Ziyuan Ma
AU  - Yixin Wu
AU  - Wenbin Liao
AU  - Feiran Zhang
AU  - Wenjie Li
AU  - Lihao Wang
AU  - Hao Wang
AU  - Xiaoqing Zheng
AU  - Xuefeng Yan
AU  - Lei Bai
AU  - Ya-Qin Zhang
AU  - Shuyi Zhang
AU  - Wei-Ying Ma
AU  - Dahua Lin
AU  - Bowen Zhou
AU  - Hao Zhou
PY  - 2026
DA  - 2026/09/24
PB  - arXiv
DO  - 10.48550/arxiv.2609.28921
UR  - https://arxiv.org/abs/2609.28921
AB  - Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational paradigms including protein language models (PLMs), large language models (LLMs), and LLM-based agents have shown promise in protein modification, their relative efficacy across realistic experimental decision-making settings remains unclear. To bridge this gap, we introduce PFArena, a benchmark comprising four controlled task interfaces that cover single-mutant generation and multi-mutant ranking. By providing varying levels of mutation fitness data, PFArena reflects four representative research scenarios characterized by differing degrees of prior experimental context. We assess six PLMs, six LLMs, and five LLM-based agents using complementary metrics to measure both peak and overall protein modification performance. Our evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data are available. Nevertheless, all model families face fundamental challenges with increasing search-space size and mutation depth. We release our code and benchmark suite to facilitate reproducible research in model-assisted protein modification.
ER  -

Abstract

Related

Cite