Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review
5 affiliations
- 1Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, China
- 2Centre for Medical Big Data and Artificial Intelligence, The First Affiliated Hospital (Southwest Hospital) of Army Medical University (Third Military Medical University), Chongqing, China
- 3Shanghai Key Laboratory of Signaling and Disease Research, School of Life Sciences and Technology, Tongji University, Shanghai, China
- 4The First Affiliated Hospital of Soochow University, Suzhou, China
- 5Department of Prevention Healthcare, The First Affiliated Hospital (Southwest Hospital) of Army Medical University (Third Military Medical University), Chongqing, China
Abstract
Large language models (LLMs) such as GPT-4 and DeepSeek have shown potential in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE) has become a critical benchmark for evaluating LLMs’ professional knowledge and reasoning ability in non-English contexts. This scoping review aims to synthesize research evaluating LLM performance on the CNMLE and related Chinese medical examinations, identifying performance trends, methodological gaps, and future directions. A scoping review was conducted in accordance with PRISMA-ScR guidelines. PubMed and Web of Science were searched on June 12, 2025, using terms related to LLMs and Chinese medical examinations. Studies were included if they evaluated any LLM on national-level Chinese medical exams and reported performance metrics. Two reviewers screened and extracted data. Study quality was assessed using a 12-item checklist covering dataset characteristics, LLM setup, and evaluation methods. Fisher’s exact test was used to assess differences in the passing rates of different LLMs. 14 studies were included, covering 51 evaluation records across 8 types of Chinese medical examinations, including CNMLE and specialty exams such as critical care, radiation oncology, and ultrasound medicine. Exam years ranged from 2017 to 2024, with a shift toward using recent exams. A total of 9 LLMs were evaluated, including GPT-3.5, GPT-4, GPT-4o, ERNIE, DeepSeek-R1, Qwen-72B, Baichuan2-7B, Baichuan2-13B and DISC-MedLLM. GPT-4o and DeepSeek-R1 achieved the highest scores of 552 (92.00%) and 523 (87.20%) in the 2024 CNMLE, respectively, both surpassing the passing threshold. In an exploratory comparison, GPT-4 achieved a higher pass rate than GPT-3.5 with a statistically significant difference, although inconsistencies in datasets, model transparency, and evaluation criteria persist across the included studies. LLMs show improving performance on Chinese medical exams, especially newer models. Nevertheless, future research should prioritize standardized benchmarks, multimodal capabilities, and transparent evaluation to ensure meaningful clinical relevance and educational value. Author summary: Large language models (LLMs) like GPT-4 and DeepSeek are emerging as powerful tools in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE), a standardized exam used to certify physicians, has become a key benchmark for evaluating whether LLMs can understand and reason through real-world clinical content in a non-English setting. In this scoping review, we analyzed 14 peer-reviewed studies that evaluated the performance of 9 different LLMs, including both GPT series and Chinese open-source LLMs, on the CNMLE and seven other types of national medical examinations in China. We found that LLMs such as GPT-4o and DeepSeek-R1 achieved high scores on recent exams, in some cases surpassing the official passing threshold. However, studies varied widely in their methods, data transparency, and evaluation criteria, highlighting the need for more standardized benchmarks and testing protocols. Our findings offer a comprehensive overview of how LLMs are currently performing on Chinese medical exams and outline the key challenges and future directions for integrating these models into medical education and licensing systems. This research provides timely insights for developers, educators, and policymakers working at the intersection of AI, healthcare, and global medical training.
Related
- Large Language Models as Simulated Candidates in Objective Structured Clinical Examinations: A Rubric-Mapping Proof-of-Concept Study
- Does low confidence drive reliance on AI, or does reliance erode confidence? A four-wave RI-CLPM study of chinese language and literature undergraduates
- COVID-19 infection and Hong Kong elementary children’s Chinese and English reading proficiencies: Parent-reported proficiency versus performance-based reading tests
- Maximal Fat Oxidation Is Associated With Moderate- and High-Speed Running Performance During Competitive Matches in Youth National-Level Football Players: A Retrospective In-Season Study
- Intrinsically disordered regions navigate DNA replication licensing factors to specific genomic loci.
BibTeX
@article{zong2026performance,
title = {{Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review}},
author = {Hui Zong and Jiao Wang and Jiaxue Cha and Rongrong Wu and Shanshan Hu and Yan Zhao and Bairong Shen},
journal = {PLOS Digital Health},
year = {2026},
month = oct,
doi = {10.1371/journal.pdig.0001773},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001773},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review
AU - Hui Zong
AU - Jiao Wang
AU - Jiaxue Cha
AU - Rongrong Wu
AU - Shanshan Hu
AU - Yan Zhao
AU - Bairong Shen
PY - 2026
DA - 2026/10/09
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001773
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001773
AB - Large language models (LLMs) such as GPT-4 and DeepSeek have shown potential in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE) has become a critical benchmark for evaluating LLMs’ professional knowledge and reasoning ability in non-English contexts. This scoping review aims to synthesize research evaluating LLM performance on the CNMLE and related Chinese medical examinations, identifying performance trends, methodological gaps, and future directions. A scoping review was conducted in accordance with PRISMA-ScR guidelines. PubMed and Web of Science were searched on June 12, 2025, using terms related to LLMs and Chinese medical examinations. Studies were included if they evaluated any LLM on national-level Chinese medical exams and reported performance metrics. Two reviewers screened and extracted data. Study quality was assessed using a 12-item checklist covering dataset characteristics, LLM setup, and evaluation methods. Fisher’s exact test was used to assess differences in the passing rates of different LLMs. 14 studies were included, covering 51 evaluation records across 8 types of Chinese medical examinations, including CNMLE and specialty exams such as critical care, radiation oncology, and ultrasound medicine. Exam years ranged from 2017 to 2024, with a shift toward using recent exams. A total of 9 LLMs were evaluated, including GPT-3.5, GPT-4, GPT-4o, ERNIE, DeepSeek-R1, Qwen-72B, Baichuan2-7B, Baichuan2-13B and DISC-MedLLM. GPT-4o and DeepSeek-R1 achieved the highest scores of 552 (92.00%) and 523 (87.20%) in the 2024 CNMLE, respectively, both surpassing the passing threshold. In an exploratory comparison, GPT-4 achieved a higher pass rate than GPT-3.5 with a statistically significant difference, although inconsistencies in datasets, model transparency, and evaluation criteria persist across the included studies. LLMs show improving performance on Chinese medical exams, especially newer models. Nevertheless, future research should prioritize standardized benchmarks, multimodal capabilities, and transparent evaluation to ensure meaningful clinical relevance and educational value. Author summary: Large language models (LLMs) like GPT-4 and DeepSeek are emerging as powerful tools in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE), a standardized exam used to certify physicians, has become a key benchmark for evaluating whether LLMs can understand and reason through real-world clinical content in a non-English setting. In this scoping review, we analyzed 14 peer-reviewed studies that evaluated the performance of 9 different LLMs, including both GPT series and Chinese open-source LLMs, on the CNMLE and seven other types of national medical examinations in China. We found that LLMs such as GPT-4o and DeepSeek-R1 achieved high scores on recent exams, in some cases surpassing the official passing threshold. However, studies varied widely in their methods, data transparency, and evaluation criteria, highlighting the need for more standardized benchmarks and testing protocols. Our findings offer a comprehensive overview of how LLMs are currently performing on Chinese medical exams and outline the key challenges and future directions for integrating these models into medical education and licensing systems. This research provides timely insights for developers, educators, and policymakers working at the intersection of AI, healthcare, and global medical training.
SN - 2767-3170
ER -