{
  "abstract": "Large language models (LLMs) such as GPT-4 and DeepSeek have shown potential in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE) has become a critical benchmark for evaluating LLMs’ professional knowledge and reasoning ability in non-English contexts. This scoping review aims to synthesize research evaluating LLM performance on the CNMLE and related Chinese medical examinations, identifying performance trends, methodological gaps, and future directions. A scoping review was conducted in accordance with PRISMA-ScR guidelines. PubMed and Web of Science were searched on June 12, 2025, using terms related to LLMs and Chinese medical examinations. Studies were included if they evaluated any LLM on national-level Chinese medical exams and reported performance metrics. Two reviewers screened and extracted data. Study quality was assessed using a 12-item checklist covering dataset characteristics, LLM setup, and evaluation methods. Fisher’s exact test was used to assess differences in the passing rates of different LLMs. 14 studies were included, covering 51 evaluation records across 8 types of Chinese medical examinations, including CNMLE and specialty exams such as critical care, radiation oncology, and ultrasound medicine. Exam years ranged from 2017 to 2024, with a shift toward using recent exams. A total of 9 LLMs were evaluated, including GPT-3.5, GPT-4, GPT-4o, ERNIE, DeepSeek-R1, Qwen-72B, Baichuan2-7B, Baichuan2-13B and DISC-MedLLM. GPT-4o and DeepSeek-R1 achieved the highest scores of 552 (92.00%) and 523 (87.20%) in the 2024 CNMLE, respectively, both surpassing the passing threshold. In an exploratory comparison, GPT-4 achieved a higher pass rate than GPT-3.5 with a statistically significant difference, although inconsistencies in datasets, model transparency, and evaluation criteria persist across the included studies. LLMs show improving performance on Chinese medical exams, especially newer models. Nevertheless, future research should prioritize standardized benchmarks, multimodal capabilities, and transparent evaluation to ensure meaningful clinical relevance and educational value. Author summary: Large language models (LLMs) like GPT-4 and DeepSeek are emerging as powerful tools in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE), a standardized exam used to certify physicians, has become a key benchmark for evaluating whether LLMs can understand and reason through real-world clinical content in a non-English setting. In this scoping review, we analyzed 14 peer-reviewed studies that evaluated the performance of 9 different LLMs, including both GPT series and Chinese open-source LLMs, on the CNMLE and seven other types of national medical examinations in China. We found that LLMs such as GPT-4o and DeepSeek-R1 achieved high scores on recent exams, in some cases surpassing the official passing threshold. However, studies varied widely in their methods, data transparency, and evaluation criteria, highlighting the need for more standardized benchmarks and testing protocols. Our findings offer a comprehensive overview of how LLMs are currently performing on Chinese medical exams and outline the key challenges and future directions for integrating these models into medical education and licensing systems. This research provides timely insights for developers, educators, and policymakers working at the intersection of AI, healthcare, and global medical training.",
  "abstract_shown": true,
  "affiliations": [
    "Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, China",
    "Centre for Medical Big Data and Artificial Intelligence, The First Affiliated Hospital (Southwest Hospital) of Army Medical University (Third Military Medical University), Chongqing, China",
    "Shanghai Key Laboratory of Signaling and Disease Research, School of Life Sciences and Technology, Tongji University, Shanghai, China",
    "The First Affiliated Hospital of Soochow University, Suzhou, China",
    "Department of Prevention Healthcare, The First Affiliated Hospital (Southwest Hospital) of Army Medical University (Third Military Medical University), Chongqing, China"
  ],
  "article_type": "Research Article",
  "arxiv": null,
  "author_total": 7,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Hui Zong"
    },
    {
      "affs": [
        1
      ],
      "name": "Jiao Wang"
    },
    {
      "affs": [
        2
      ],
      "name": "Jiaxue Cha"
    },
    {
      "affs": [
        3
      ],
      "name": "Rongrong Wu"
    },
    {
      "affs": [
        4
      ],
      "name": "Shanshan Hu"
    },
    {
      "name": "Yan Zhao"
    },
    {
      "name": "Bairong Shen"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/7ctw0712afhca.bib",
  "cited_by": 0,
  "doi": "10.1371/journal.pdig.0001773",
  "fulltext_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001773",
  "id": "7ctw0712afhca",
  "issns": [
    "2767-3170"
  ],
  "landing_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001773",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ],
    [
      "plos",
      "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Language"
    ],
    [
      "plos",
      "/Biology and life sciences/Psychology/Cognitive psychology/Language"
    ],
    [
      "plos",
      "/Medicine and health sciences"
    ],
    [
      "plos",
      "/Medicine and health sciences/Clinical medicine"
    ],
    [
      "plos",
      "/Medicine and health sciences/Critical care and emergency medicine"
    ],
    [
      "plos",
      "/Medicine and health sciences/Medical humanities/Medical education"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology"
    ],
    [
      "plos",
      "/People and places/Geographical locations/Asia/China"
    ],
    [
      "plos",
      "/Research and analysis methods/Database and informatics methods/Database searching"
    ],
    [
      "plos",
      "/Social sciences/Psychology/Cognitive psychology/Language"
    ],
    [
      "plos",
      "/Social sciences/Sociology/Education/Medical education"
    ]
  ],
  "openalex": "W7221192868",
  "pdf_url": "https://journals.plos.org/digitalhealth/article/file?id=10.1371/journal.pdig.0001773&type=printable",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-09",
  "publisher": "Public Library of Science",
  "ris": "https://research.draconitic.com/paper/7ctw0712afhca.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:30Z",
      "native_id": "10.1371/journal.pdig.0001773",
      "source": "plos",
      "url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001773"
    }
  ],
  "subjects": [
    {
      "field": "biology",
      "origin": "plos:neuroscience",
      "subfield": "neuroscience"
    },
    {
      "field": "medicine",
      "origin": "plos:oncology",
      "subfield": "oncology"
    }
  ],
  "title": "Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review",
  "title_text": "Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T12:17:30Z",
  "url": "https://research.draconitic.com/paper/7ctw0712afhca",
  "venue": "PLOS Digital Health",
  "versions": []
}