Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions
- 1Department of Kaumarbhritya, Mahatma Gandhi Ayurved Hospital, Salod (H), Wardha, Maharashtra 442001 India
Abstract
Background: Large language models (LLMs), particularly successive iterations of ChatGPT (GPT-3.5 and GPT-4), have demonstrated increasingly sophisticated performance on standardised medical assessments and are now widely accessible to undergraduate medical students worldwide. Although the evidence base has expanded rapidly since ChatGPT’s public release in November 2022, findings remain fragmented across three largely distinct research domains: examination performance, knowledge acquisition, and learner perceptions.
Methods: A PRISMA 2020-compliant systematic review was conducted. Structured searches were performed in PubMed (MEDLINE), Scopus, and Web of Science Core Collection (January 2022–March 2026), supplemented by grey literature and citation tracking. Eligible studies examined ChatGPT (GPT-3.5, GPT-4, GPT-4o) or comparable transformer-based LLMs in undergraduate medical educational contexts. Two reviewers independently screened records, extracted data, and assessed risk of bias (Newcastle-Ottawa Scale; ROBINS-I). Narrative synthesis followed the Synthesis Without Meta-analysis (SWiM) reporting guideline.
Results: Of 1,847 records identified, 43 studies (approximately 12,400 participants across 19 countries) met inclusion criteria. GPT-4 achieved passing scores on the United States Medical Licensing Examination (USMLE) Steps 1–3 in all studies in which it was tested (mean accuracy 86%; IQR 81%--89%), significantly exceeding GPT-3.5 performance (mean 56%). Accuracy declined on multi-step clinical vignettes (61%–71%) and image-dependent items (48%–63%). Six of twelve studies reported modest short-term knowledge acquisition benefits effect sizes ranged from 0.31 to 0.71 (median⁓0.53) across six studies without durable retention. Students broadly endorsed LLMs as educational aids (74% positive perception); however, 61% raised academic integrity concerns, 59% cited hallucination risk, and 82% reported receiving no institutional guidance. Faculty expressed more tempered views (51% positive). Risk of bias was moderate-to-high in 86% of included studies.
Conclusions: GPT-4 reaches examination-passing thresholds on major medical licensing assessments but demonstrates clinically important limitations in multi-step reasoning, uncertainty calibration, and multimodal task performance. The near-universal absence of institutional guidance represents the most urgent and actionable finding. Longitudinal controlled research with validated outcome measures and development of standardised AI literacy curricula are the field’s most pressing priorities.
Supplementary Information: The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-09999-7.
Related
- A Systematic Review of Spaced Repetition, Retrieval Practice, and Academic Performance in Undergraduate Medical Students
- Perceptions and Experiences of the Foster Family Program Among Alumni and Undergraduate Medical Students at CMC Ludhiana: An Exploratory Mixed-Methods Study
- Assessment of Knowledge, Attitudes, and Perceptions Regarding Artificial Intelligence Among Undergraduate Medical Students.
- Academic performance of large language models across an undergraduate physiotherapy curriculum: a subset analysis
- Peer Evaluation Rating Behavior in Differentiating Student Performance in Undergraduate Medical Education: An Observational Study
BibTeX
@article{bodke2026large,
title = {{Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions}},
author = {Parag Bodke and Rahul Jumle},
journal = {BMC Medical Education},
year = {2026},
month = jul,
doi = {10.1186/s12909-026-09999-7},
url = {https://doi.org/10.1186/s12909-026-09999-7},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions
AU - Parag Bodke
AU - Rahul Jumle
PY - 2026
DA - 2026/07/30
JO - BMC Medical Education
DO - 10.1186/s12909-026-09999-7
UR - https://doi.org/10.1186/s12909-026-09999-7
AB - Background: Large language models (LLMs), particularly successive iterations of ChatGPT (GPT-3.5 and GPT-4), have demonstrated increasingly sophisticated performance on standardised medical assessments and are now widely accessible to undergraduate medical students worldwide. Although the evidence base has expanded rapidly since ChatGPT’s public release in November 2022, findings remain fragmented across three largely distinct research domains: examination performance, knowledge acquisition, and learner perceptions. Methods: A PRISMA 2020-compliant systematic review was conducted. Structured searches were performed in PubMed (MEDLINE), Scopus, and Web of Science Core Collection (January 2022–March 2026), supplemented by grey literature and citation tracking. Eligible studies examined ChatGPT (GPT-3.5, GPT-4, GPT-4o) or comparable transformer-based LLMs in undergraduate medical educational contexts. Two reviewers independently screened records, extracted data, and assessed risk of bias (Newcastle-Ottawa Scale; ROBINS-I). Narrative synthesis followed the Synthesis Without Meta-analysis (SWiM) reporting guideline. Results: Of 1,847 records identified, 43 studies (approximately 12,400 participants across 19 countries) met inclusion criteria. GPT-4 achieved passing scores on the United States Medical Licensing Examination (USMLE) Steps 1–3 in all studies in which it was tested (mean accuracy 86%; IQR 81%--89%), significantly exceeding GPT-3.5 performance (mean 56%). Accuracy declined on multi-step clinical vignettes (61%–71%) and image-dependent items (48%–63%). Six of twelve studies reported modest short-term knowledge acquisition benefits effect sizes ranged from 0.31 to 0.71 (median⁓0.53) across six studies without durable retention. Students broadly endorsed LLMs as educational aids (74% positive perception); however, 61% raised academic integrity concerns, 59% cited hallucination risk, and 82% reported receiving no institutional guidance. Faculty expressed more tempered views (51% positive). Risk of bias was moderate-to-high in 86% of included studies. Conclusions: GPT-4 reaches examination-passing thresholds on major medical licensing assessments but demonstrates clinically important limitations in multi-step reasoning, uncertainty calibration, and multimodal task performance. The near-universal absence of institutional guidance represents the most urgent and actionable finding. Longitudinal controlled research with validated outcome measures and development of standardised AI literacy curricula are the field’s most pressing priorities. Supplementary Information: The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-09999-7.
SN - 1472-6920
ER -