Enhancing medical education with AI: A comparative study of various large language models in radiology
- 1Department of Internal Medicine, Division of Radiology, Faculty of Medicine, King Abdulaziz University, Rabigh, Saudi Arabia
- 2Radiology Division, Department of Medicine, Faculty of Medicine, King Abdulaziz University, Rabigh, Saudi Arabia
Abstract
Objective: To compare how various Large Language Models (LLMs) perform when solving text- and image-based medical student-level radiology multiple-choice questions (MCQs). Methods: We evaluated the accuracy of seven LLMs (i.e., ChatGPT-4o, ChatGPT-5, ChatGPT-5 Plus, DeepSeek-V3.1-Terminus, Microsoft Copilot, Claude 4, and Gemini 2.5 Flash) in solving 60 text-based and 60 image-based MCQs. We compared the seven LLMs’ accuracy across question types, domains, and topics. Results: The mean accuracy across the seven evaluated LLMs was 79.9% (671/840 correct responses). Claude 4 achieved the highest observed accuracy at 86.7%, whereas Gemini 2.5 Flash and Microsoft Copilot recorded the lowest accuracy at 75.0% each. Accuracy was substantially lower for image-based questions than for text-based questions (64.5% vs. 95.2%). In the generalized estimating equations analysis, question type was significantly associated with response accuracy (Wald χ2(1) = 64.53, p < 0.001), as were question domain (Wald χ2(1) = 6.86, p = 0.009) and the question type × domain interaction (Wald χ2(3) = 80.26, p < 0.001). The omnibus effect of model was not statistically significant after accounting for these factors (Wald χ2(6) = 6.59, p = 0.361). Although Claude 4 achieved the highest observed accuracy, no pairwise model comparison remained statistically significant after Bonferroni correction. Conclusions: Large Language Models are less accurate at solving image-based than text-based radiology questions, supporting their role as adjunctive educational tools that require faculty oversight rather than independent replacements for radiology instruction.
Related
- Impression-regularized finetuning enables small language models to extract cervical spine fractures from radiology reports
- Enhancing medical image segmentation with HCE-Net: A heterogeneous feature collaborative extraction network
- Enhancing engineering competencies through Outcome-based Education: A mixed-methods study in an intelligent control course
- Telemedicine education for medical students and residents worldwide: A scoping review protocol
- Rethinking frailty and intrinsic capacity in medical education in Europe through an EAMA student collaboration
BibTeX
@article{bakhsh2026enhancing,
title = {{Enhancing medical education with AI: A comparative study of various large language models in radiology}},
author = {Noha Bakhsh and Ghofran A. Ageely},
journal = {PLOS One},
year = {2026},
month = nov,
doi = {10.1371/journal.pone.0360928},
url = {https://doi.org/10.1371/journal.pone.0360928},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Enhancing medical education with AI: A comparative study of various large language models in radiology
AU - Noha Bakhsh
AU - Ghofran A. Ageely
PY - 2026
DA - 2026/11/06
JO - PLOS One
DO - 10.1371/journal.pone.0360928
UR - https://doi.org/10.1371/journal.pone.0360928
AB - Objective: To compare how various Large Language Models (LLMs) perform when solving text- and image-based medical student-level radiology multiple-choice questions (MCQs). Methods: We evaluated the accuracy of seven LLMs (i.e., ChatGPT-4o, ChatGPT-5, ChatGPT-5 Plus, DeepSeek-V3.1-Terminus, Microsoft Copilot, Claude 4, and Gemini 2.5 Flash) in solving 60 text-based and 60 image-based MCQs. We compared the seven LLMs’ accuracy across question types, domains, and topics. Results: The mean accuracy across the seven evaluated LLMs was 79.9% (671/840 correct responses). Claude 4 achieved the highest observed accuracy at 86.7%, whereas Gemini 2.5 Flash and Microsoft Copilot recorded the lowest accuracy at 75.0% each. Accuracy was substantially lower for image-based questions than for text-based questions (64.5% vs. 95.2%). In the generalized estimating equations analysis, question type was significantly associated with response accuracy (Wald χ2(1) = 64.53, p < 0.001), as were question domain (Wald χ2(1) = 6.86, p = 0.009) and the question type × domain interaction (Wald χ2(3) = 80.26, p < 0.001). The omnibus effect of model was not statistically significant after accounting for these factors (Wald χ2(6) = 6.59, p = 0.361). Although Claude 4 achieved the highest observed accuracy, no pairwise model comparison remained statistically significant after Bonferroni correction. Conclusions: Large Language Models are less accurate at solving image-based than text-based radiology questions, supporting their role as adjunctive educational tools that require faculty oversight rather than independent replacements for radiology instruction.
SN - 1932-6203
ER -