Benchmarking AI scientists for omics data–driven biological discovery
4 affiliations
- 1MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,
- 2School of Software Engineering, Beijing Jiaotong University , Beijing, 100044,
- 3Department of Physiology and Pathophysiology, School of Basic Medical Sciences, Capital Medical University , Beijing, 100069,
- 4Center for Synthetic and Systems Biology, School of Life Sciences and School of Medicine, Tsinghua University , Beijing, 100084,
Abstract
MOTIVATION: Recent advances in large language models have enabled the emergence of AI scientists that aim to autonomously analyze biological data and assist scientific discovery. Despite rapid progress, it remains unclear to what extent these systems can extract meaningful biological insights from real experimental data. Existing benchmarks either evaluate reasoning in the absence of data or focus on predefined analytical outputs, failing to reflect realistic, data-driven biological research. RESULTS: Here, we introduce BAISBench (Biological AI Scientist Benchmark), a benchmark for evaluating AI scientists on real single-cell transcriptomic datasets. BAISBench comprises two tasks: cell type annotation across 15 expert-labeled datasets, and scientific discovery through 193 multiple-choice questions derived from biological conclusions reported in 41 published single-cell studies. We evaluated several representative AI scientists using BAISBench and, to provide a human performance baseline, invited five graduate-level bioinformaticians to collectively complete the same tasks. The results show that while current AI scientists fall short of fully autonomous biological discovery, they already demonstrate substantial potential in supporting data-driven biological research. These results position BAISBench as a practical benchmark for characterizing the current capabilities and limitations of AI scientists in biological research. We expect BAISBench to serve as a practical evaluation framework for guiding the development of more capable AI scientists and for helping biologists identify AI systems that can effectively support real-world research workflows. AVAILABILITY AND IMPLEMENTATION: https://github.com/EperLuo/BAISBench, https://huggingface.co/datasets/EperLuo/BaisBench.
Related
- From Prediction to Mechanism: Explainable AI for Plant–Microbiome Multi-Omics Integration and Biological Discovery
- Benchmarking Test Statistics and False Discovery Rate Procedures for Single-Cell Differential Expression
- Benchmarking graph-based models for in-silico toxicity prediction in drug discovery
- AI-integrated iPSC multi-omics: advancing phenotype discovery and drug target identification in rare diseases
- Multi-omics perspectives on mitochondrial vulnerability in the aging liver: biological modules and translational implications
BibTeX
@article{luo2026benchmarking,
title = {{Benchmarking AI scientists for omics data–driven biological discovery}},
author = {Erpai Luo and Jinmeng Jia and Yifan Xiong and Xiangyu Li and Xiaobo Guo and Baoqi Yu and Minsheng Hao and Lei Wei and Xuegong Zhang},
journal = {Bioinformatics},
year = {2026},
month = jul,
doi = {10.1093/bioinformatics/btag227},
url = {https://doi.org/10.1093/bioinformatics/btag227},
note = {CC BY},
}
RIS
TY - JOUR
TI - Benchmarking AI scientists for omics data–driven biological discovery
AU - Erpai Luo
AU - Jinmeng Jia
AU - Yifan Xiong
AU - Xiangyu Li
AU - Xiaobo Guo
AU - Baoqi Yu
AU - Minsheng Hao
AU - Lei Wei
AU - Xuegong Zhang
PY - 2026
DA - 2026/07/01
JO - Bioinformatics
DO - 10.1093/bioinformatics/btag227
UR - https://doi.org/10.1093/bioinformatics/btag227
AB - MOTIVATION: Recent advances in large language models have enabled the emergence of AI scientists that aim to autonomously analyze biological data and assist scientific discovery. Despite rapid progress, it remains unclear to what extent these systems can extract meaningful biological insights from real experimental data. Existing benchmarks either evaluate reasoning in the absence of data or focus on predefined analytical outputs, failing to reflect realistic, data-driven biological research. RESULTS: Here, we introduce BAISBench (Biological AI Scientist Benchmark), a benchmark for evaluating AI scientists on real single-cell transcriptomic datasets. BAISBench comprises two tasks: cell type annotation across 15 expert-labeled datasets, and scientific discovery through 193 multiple-choice questions derived from biological conclusions reported in 41 published single-cell studies. We evaluated several representative AI scientists using BAISBench and, to provide a human performance baseline, invited five graduate-level bioinformaticians to collectively complete the same tasks. The results show that while current AI scientists fall short of fully autonomous biological discovery, they already demonstrate substantial potential in supporting data-driven biological research. These results position BAISBench as a practical benchmark for characterizing the current capabilities and limitations of AI scientists in biological research. We expect BAISBench to serve as a practical evaluation framework for guiding the development of more capable AI scientists and for helping biologists identify AI systems that can effectively support real-world research workflows. AVAILABILITY AND IMPLEMENTATION: https://github.com/EperLuo/BAISBench, https://huggingface.co/datasets/EperLuo/BaisBench.
SN - 1367-4803
SN - 1367-4811
ER -