Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases
19 affiliations
- 1Department of Biomedical Engineering, The Chinese University of Hong Kong
- 2Department of Interventional Radiology, Beijing Friendship Hospital, Capital Medical University
- 3Department of Metabolism, Digestion and Reproduction, Imperial College London
- 4Department of Radiology, Qinghai University Affiliated Hospital, Qinghai University
- 5Institute of Ophthalmology, University College London
- 6Department of Hepatobiliary Surgery, Xijing Hospital, Fourth Military Medical University
- 7State Key Laboratory of Pathogenesis, Prevention and Treatment of High Incidence Diseases in Central Asia, Clinical Medicine Institute, The First Affiliated Hospital of Xinjiang Medical University
- 8Department of Ophthalmology, Beijing Friendship Hospital, Capital Medical University
- 9Department of Anthropology, The Chinese University of Hong Kong
- 10Artificial Intelligence Thrust, The Hong Kong University of Science and Technology (HKUST)
- 11The Hamlyn Centre for Robotic Surgery, Imperial College London
- 12Department of Cardiology, The 7th People’s Hospital of Zhengzhou
- 13Department of Cardiology, The First Affiliated Hospital, Sun-yat Sen University
- 14Department of Cardiology, China-Japan Union Hospital of Jilin University
- 15Department of Hepatobiliary and Pancreatic Surgery, Qinghai University Affiliated Hospital
- 16Department of Radiology, Lanzhou University Second Hospital
- 17Department of Cardiology, Zhongshan Hospital, Fudan University
- 18Department of Medical Imaging, Henan Provincial People’s Hospital
- 19Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University
Abstract
While multimodal large language models (LLMs) demonstrate significant potential in healthcare applications, their clinical utility is difficult to appraise. Current evaluations of medical-assisting LLMs are often limited by sparse human expertise, narrow specialty scope, and reliance on multiple-choice benchmarks or synthetic vignettes, which can inflate performance and obscure clinical utility. We conducted a multicenter, multidisciplinary study in which more than 400 physicians—spanning seven specialties, varied experience levels, and multiple geographic settings—evaluated LLM-generated free-text responses to real, de-identified clinical cases. In a matched-control design, we also deployed an equivalent number of AI agents configured to mirror physician characteristics to examine whether automated evaluators can supplement or replace human assessment. Our results demonstrated that physician assessments exhibited substantial heterogeneity by clinical seniority and practice environment, leading to notable shifts in relative model rankings across cohorts. While AI agents delivered highly efficient, directionally aligned assessments, they did not fully capture the nuances of human clinical judgment and could not substitute for physician-centered evaluation. Instead, they promise assistive tools that can triage or pre-screen outputs to reduce human burden.
Related
- The Influence of Large Language Model-Generated Artificial Intelligence Risk Estimates on Clinical Decision-Making in Breast Imaging: A Simulation-Based Study
- Stakeholder-Reported Requirements for Future Artificial Intelligence Decision Support in Periodontal Care: A Cross-Sectional Survey of Dentists, Physicians and Patients in Kazakhstan.
- Quality of physicians responses using a conversational artificial intelligence system: a randomized vignette experiment in Latin America
- Evaluating the impact of social determinants of health on model performance and fairness in artificial intelligence for glaucoma progression prediction
- Artificial intelligence awareness, usage, and readiness among fellows of the West African college of physicians.
BibTeX
@article{shi2026physicians,
title = {{Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases}},
author = {Peilun Shi and Jian Li and Ziqi Yang and Yuntai Cao and Lie Ju and Haihua Bao and Hao Xu and Yichen Fan and Tianmin Ren and Yiyang Xiao and Jinjing Zhu and Junhong Chen and Yong Dong and Xun Hu and Yaliang Tong and Zhan Wang and Yuchen Xiang and Jinshi Zhao and Junlin Zhou and Burak Temelkuran and Yukun Zhou and Long Jin and Chenguang Li and Meiyun Wang and Hui Xiong and Pearse A. Keane and Junbo Ge and Ningli Wang},
journal = {npj Digital Medicine},
year = {2026},
month = jul,
doi = {10.1038/s41746-026-02942-6},
url = {https://doi.org/10.1038/s41746-026-02942-6},
note = {CC BY},
}
RIS
TY - JOUR
TI - Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases
AU - Peilun Shi
AU - Jian Li
AU - Ziqi Yang
AU - Yuntai Cao
AU - Lie Ju
AU - Haihua Bao
AU - Hao Xu
AU - Yichen Fan
AU - Tianmin Ren
AU - Yiyang Xiao
AU - Jinjing Zhu
AU - Junhong Chen
AU - Yong Dong
AU - Xun Hu
AU - Yaliang Tong
AU - Zhan Wang
AU - Yuchen Xiang
AU - Jinshi Zhao
AU - Junlin Zhou
AU - Burak Temelkuran
AU - Yukun Zhou
AU - Long Jin
AU - Chenguang Li
AU - Meiyun Wang
AU - Hui Xiong
AU - Pearse A. Keane
AU - Junbo Ge
AU - Ningli Wang
PY - 2026
DA - 2026/07/01
JO - npj Digital Medicine
DO - 10.1038/s41746-026-02942-6
UR - https://doi.org/10.1038/s41746-026-02942-6
AB - While multimodal large language models (LLMs) demonstrate significant potential in healthcare applications, their clinical utility is difficult to appraise. Current evaluations of medical-assisting LLMs are often limited by sparse human expertise, narrow specialty scope, and reliance on multiple-choice benchmarks or synthetic vignettes, which can inflate performance and obscure clinical utility. We conducted a multicenter, multidisciplinary study in which more than 400 physicians—spanning seven specialties, varied experience levels, and multiple geographic settings—evaluated LLM-generated free-text responses to real, de-identified clinical cases. In a matched-control design, we also deployed an equivalent number of AI agents configured to mirror physician characteristics to examine whether automated evaluators can supplement or replace human assessment. Our results demonstrated that physician assessments exhibited substantial heterogeneity by clinical seniority and practice environment, leading to notable shifts in relative model rankings across cohorts. While AI agents delivered highly efficient, directionally aligned assessments, they did not fully capture the nuances of human clinical judgment and could not substitute for physician-centered evaluation. Instead, they promise assistive tools that can triage or pre-screen outputs to reduce human burden.
SN - 2398-6352
ER -