Real Science Is Harder Than Benchmarks: Evaluating Advanced AI Frameworks on Published Studies. II. Antibody Properties, Lipid-RNA Interactions
- 1ML LC
Abstract
Artificial Intelligence (AI) frameworks for automating scientific research have shown strong performance on benchmarks, but their utility for real-world industrial research remains insufficiently characterized. Extending the analysis presented in the first paper of this series, we evaluated the same five advanced AI research frameworks (Kosmos, K-Dense, ToolUniverse, BioAgents from bio.xyz, and the AI Scientist-v2 from Sakana AI) on two more projects of high practical importance for biopharmaceutical development: predicting antibody developability properties with the use of pretrained protein language model embeddings, and modeling non-covalent lipid-RNA interactions in lipid nanoparticles with all-atom molecular dynamics (MD) simulations. The AI frameworks again showed genuine strengths, including unprompted identification of subtle methodological issues, successful use of pretrained protein embeddings, and consistent reporting of p-values and confidence intervals often absent from the original papers. However, no framework approached the scope of the original studies, and severe failures and hallucinations were observed. Our results confirm and extend the conclusion of the first paper that real published research from pharmaceutical companies that we tried to reproduce proved to be considerably harder for current AI frameworks than standard benchmarks suggest.
Related
- corrdeconv corrects deconvolution's misreading of disease cell states as abundance, overturning benchmarks and outperforming every published deconvolution method
- Evaluating Autonomous LLM Agents Across Molecular Prediction and Optimization Benchmarks
- Evaluating the Robustness of Path-Preservation Benchmarks for Dimensionality Reduction Across Point-Density Thresholds: Linear and Cyclic Single-Cell Trajectories
- Interactions between key bacterial microbiota and soil properties in healthy soils suppress tobacco bacterial wilt
- VOC·BIO Library: An Integrated Platform for Published Aphid GC-MS Evidence, Biological Contexts, Cheminformatics and Exploratory OBP Docking
BibTeX
@misc{bhutada2026real,
title = {{Real Science Is Harder Than Benchmarks: Evaluating Advanced AI Frameworks on Published Studies. II. Antibody Properties, Lipid-RNA Interactions}},
author = {P. Bhutada and N. Goyal and T. K. Lakhankiya and S. D. Narahari and S. S. N. Thangaraju and T. S. Nayak and Y. Peng and G. S. A. Thota and R. S. D. Thota and Z. Wang and K. Lee and A. Sinitskiy},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.03.749176},
url = {https://www.biorxiv.org/content/10.64898/2026.09.03.749176v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Real Science Is Harder Than Benchmarks: Evaluating Advanced AI Frameworks on Published Studies. II. Antibody Properties, Lipid-RNA Interactions
AU - P. Bhutada
AU - N. Goyal
AU - T. K. Lakhankiya
AU - S. D. Narahari
AU - S. S. N. Thangaraju
AU - T. S. Nayak
AU - Y. Peng
AU - G. S. A. Thota
AU - R. S. D. Thota
AU - Z. Wang
AU - K. Lee
AU - A. Sinitskiy
PY - 2026
DA - 2026/09/03
PB - bioRxiv
DO - 10.64898/2026.09.03.749176
UR - https://www.biorxiv.org/content/10.64898/2026.09.03.749176v1
AB - Artificial Intelligence (AI) frameworks for automating scientific research have shown strong performance on benchmarks, but their utility for real-world industrial research remains insufficiently characterized. Extending the analysis presented in the first paper of this series, we evaluated the same five advanced AI research frameworks (Kosmos, K-Dense, ToolUniverse, BioAgents from bio.xyz, and the AI Scientist-v2 from Sakana AI) on two more projects of high practical importance for biopharmaceutical development: predicting antibody developability properties with the use of pretrained protein language model embeddings, and modeling non-covalent lipid-RNA interactions in lipid nanoparticles with all-atom molecular dynamics (MD) simulations. The AI frameworks again showed genuine strengths, including unprompted identification of subtle methodological issues, successful use of pretrained protein embeddings, and consistent reporting of p-values and confidence intervals often absent from the original papers. However, no framework approached the scope of the original studies, and severe failures and hallucinations were observed. Our results confirm and extend the conclusion of the first paper that real published research from pharmaceutical companies that we tried to reproduce proved to be considerably harder for current AI frameworks than standard benchmarks suggest.
ER -