VibeProteinBench: An Evaluation Benchmark for Language-interfaced Vibe Protein Design
Abstract
Protein design aims to compose amino-acid sequences that fold into stable three-dimensional structures while satisfying targeted functional properties. The field is increasingly shifting toward vibe protein design, where a single model is expected to generate novel sequences, engineer existing proteins, and reason about protein characteristics through flexible natural-language constraints. However, existing benchmarks typically evaluate isolated aspects of protein design or assume predefined structured inputs, making them ill-suited to assess this broad, open-ended setting. To address this gap, we present Vibe Protein Design Benchmark (VibeProteinBench), a language-interfaced benchmark that evaluates whether a single model can operate across three complementary stages of a computational protein-design workflow: recognition, engineering, and generation. Each stage is grounded in expert-curated mechanistic rationales and multi-faceted in silico validation, to computationally verify whether model outputs are biologically plausible. Across diverse general-purpose and domain-specialized LLMs, evaluated with and without tool access, no model performs strongly across all three stages at once. Beyond measuring end-task success, VibeProteinBench exposes where current systems break: we identify recurring failure patterns and characterize how tool-augmented agents select tool arguments. Together, these findings show that vibe protein design remains a substantial open challenge.
Related
- Comprehensive Evaluation of Protein Language Model Embeddings for Drug-Target Affinity Prediction
- Protify: A Benchmarking Platform Revealing Task Dependence and Homology Leakage in Protein Language Model Evaluation
- OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models
- A Comparative Benchmark of Biomedical Language Models for Concept Normalization from Real-World Text
- Do Vision CIL Methods Transfer to Protein? A Diagnostic Benchmark of Frozen Protein Encoders
BibTeX
@misc{seo2026vibeproteinbench,
title = {{VibeProteinBench: An Evaluation Benchmark for Language-interfaced Vibe Protein Design}},
author = {Hyunjin Seo and Hongjoon Ahn and Jimin Park and Sungjun Han and Gyubok Lee and Soojung Yang and Cheng-Hao Liu and Joseph S Brown and Leo Chen and Gina El Nesr and Feyisayo Eweje and Sarah Gurev and Hyejin Lee and Junlang Liu and Zhihui Qi and Gyu Rie Lee and Sungsoo Ahn and Jamin Shin and Sangwon Jung},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.10978},
eprint = {2605.10978},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.10978},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - VibeProteinBench: An Evaluation Benchmark for Language-interfaced Vibe Protein Design
AU - Hyunjin Seo
AU - Hongjoon Ahn
AU - Jimin Park
AU - Sungjun Han
AU - Gyubok Lee
AU - Soojung Yang
AU - Cheng-Hao Liu
AU - Joseph S Brown
AU - Leo Chen
AU - Gina El Nesr
AU - Feyisayo Eweje
AU - Sarah Gurev
AU - Hyejin Lee
AU - Junlang Liu
AU - Zhihui Qi
AU - Gyu Rie Lee
AU - Sungsoo Ahn
AU - Jamin Shin
AU - Sangwon Jung
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2605.10978
UR - https://arxiv.org/abs/2605.10978
AB - Protein design aims to compose amino-acid sequences that fold into stable three-dimensional structures while satisfying targeted functional properties. The field is increasingly shifting toward vibe protein design, where a single model is expected to generate novel sequences, engineer existing proteins, and reason about protein characteristics through flexible natural-language constraints. However, existing benchmarks typically evaluate isolated aspects of protein design or assume predefined structured inputs, making them ill-suited to assess this broad, open-ended setting. To address this gap, we present Vibe Protein Design Benchmark (VibeProteinBench), a language-interfaced benchmark that evaluates whether a single model can operate across three complementary stages of a computational protein-design workflow: recognition, engineering, and generation. Each stage is grounded in expert-curated mechanistic rationales and multi-faceted in silico validation, to computationally verify whether model outputs are biologically plausible. Across diverse general-purpose and domain-specialized LLMs, evaluated with and without tool access, no model performs strongly across all three stages at once. Beyond measuring end-task success, VibeProteinBench exposes where current systems break: we identify recurring failure patterns and characterize how tool-augmented agents select tool arguments. Together, these findings show that vibe protein design remains a substantial open challenge.
ER -