Benchmarking and developing large language models using one million clinical trials
5 affiliations
- 1Keiji AI, Seattle, WA, USA
- 2School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL, USA
- 3Division of Intramural Research, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
- 4Health Data Research UK, London, UK
- 5Usher Institute, Edinburgh Medical School, University of Edinburgh, Edinburgh, UK
Abstract
Developing artificial intelligence (AI) for clinical research requires a comprehensive data foundation for model benchmarking and development. Here, we introduce , a large-scale structured resource aggregating 1.6M clinical trial records from fifteen global registries linked with biomedical ontologies and literature. Using this resource, we construct 152K training and testing samples spanning eight clinical research tasks, including systematic review, trial design, and trial optimization. Benchmarking cutting-edge large language models (LLMs) reveals limited clinical reasoning capability in generic LLMs. In contrast, an 8B LLM developed on using supervised fine-tuning and reinforcement learning outperforms 70B generic counterparts across all eight tasks, with relative improvements of 73.7, 67.6, 38.4, 37.8, 26.5, 20.7, 20.0, 18.1, and 5.2%, respectively. These results demonstrate the potential of domain-adapted AI to improve evidence synthesis and clinical trial design, establishing as a foundation for scaling AI in clinical research.
Related
- BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
- ClinStigma: A Clinician-Guided Framework for Developing Synthetic Benchmarks of Contextual Stigmatizing Language in Clinical Documentation
- Large language models detect methodological errors that human peer review misses in clinical trials
- EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models
- Variability among large language models in assessing CONSORT compliance of published randomized clinical trials
BibTeX
@article{wang2026benchmarking,
title = {{Benchmarking and developing large language models using one million clinical trials}},
author = {Zifeng Wang and Jiacheng Lin and Qiao Jin and Junyi Gao and Jathurshan Pradeepkumar and Pengcheng Jiang and Zhiyong Lu and Jimeng Sun},
journal = {npj Digital Medicine},
year = {2026},
month = jul,
doi = {10.1038/s41746-026-02933-7},
url = {https://doi.org/10.1038/s41746-026-02933-7},
note = {CC BY},
}
RIS
TY - JOUR
TI - Benchmarking and developing large language models using one million clinical trials
AU - Zifeng Wang
AU - Jiacheng Lin
AU - Qiao Jin
AU - Junyi Gao
AU - Jathurshan Pradeepkumar
AU - Pengcheng Jiang
AU - Zhiyong Lu
AU - Jimeng Sun
PY - 2026
DA - 2026/07/31
JO - npj Digital Medicine
DO - 10.1038/s41746-026-02933-7
UR - https://doi.org/10.1038/s41746-026-02933-7
AB - Developing artificial intelligence (AI) for clinical research requires a comprehensive data foundation for model benchmarking and development. Here, we introduce , a large-scale structured resource aggregating 1.6M clinical trial records from fifteen global registries linked with biomedical ontologies and literature. Using this resource, we construct 152K training and testing samples spanning eight clinical research tasks, including systematic review, trial design, and trial optimization. Benchmarking cutting-edge large language models (LLMs) reveals limited clinical reasoning capability in generic LLMs. In contrast, an 8B LLM developed on using supervised fine-tuning and reinforcement learning outperforms 70B generic counterparts across all eight tasks, with relative improvements of 73.7, 67.6, 38.4, 37.8, 26.5, 20.7, 20.0, 18.1, and 5.2%, respectively. These results demonstrate the potential of domain-adapted AI to improve evidence synthesis and clinical trial design, establishing as a foundation for scaling AI in clinical research.
SN - 2398-6352
ER -