Performance of Contemporary Large Language Models on the National Prosthodontic Resident Examination: A Multi-Year Comparative Study
- 1Tufts University School of Dental Medicine
Abstract
PurposeTo compare the performance of three large language models (LLMs)--ChatGPT-o1, Grok 3, and Gemini 2.0 Pro--on the American College of Prosthodontists National Prosthodontic Resident Examination (NPRE).
Materials and MethodsOfficial NPRE examinations from years 2015-2024 were retrieved from the American College of Prosthodontists website and used to prompt each LLM. Paid access tiers and advanced reasoning were enabled. Responses from LLMs were recorded and compared against the official answer keys for accuracy, variability across years, and consistency of responses. Duplicate questions were retained for the primary analysis and removed in a sensitivity analysis. Accuracy (percent correct) was compared using generalized estimating equations binary logistic regression (primary); Cochrans Q with Bonferroni-adjusted McNemar tests (sensitivity). Year-to-year variability (SD of annual accuracy) was compared using weighted Levene tests. Consistency across duplicate items was quantified as the percentage of perfect matches and compared using Cochrans Q with Bonferroni-adjusted McNemar tests.
ResultsAcross 1,487 questions, overall accuracy differed among models (global P < 0.001). ChatGPT-o1 achieved 78.4% correct, Grok 3 76.4%, and Gemini 2.0 Pro 73.2%; ChatGPT-o1 and Grok 3s accuracy significantly exceeded Gemini 2.0 Pro (both P [≤] 0.016), with no significant difference between ChatGPT-o1 and Grok 3 (P = 0.099). After removing duplicates (n = 1,200), results were similar: ChatGPT-o1, Grok 3, and Gemini 2.0 Pro were 77.1%, 76.6%, and 71.5% correct, respectively; ChatGPT-o1 and Grok 3s accuracy was significantly higher than Gemini 2.0 Pro (both P < 0.001). Variability across years did not differ significantly (P [≥] 0.097). Grok 3 showed significantly higher perfect-match consistency (79.2%) than ChatGPT-o1 (69.4%) and Gemini 2.0 Pro (64.5%) (P [≤] 0.010).
ConclusionsContemporary LLMs showed 70-80% accuracy on postgraduate prosthodontic multiple-choice content. While promising for educational support, residual error and inconsistency warrant human oversight and transparent methods reporting before curricular adoption.
Related
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
- Expert-Guided Visual Correction for Characterizing Diagnostic Performance and Error Patterns of Multimodal Large Language Models Using Periodontal In-Service Examination Images
- Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination
- Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions
- Performance of large language models in prescription review: a comparative study
BibTeX
@misc{chen2026performance,
title = {{Performance of Contemporary Large Language Models on the National Prosthodontic Resident Examination: A Multi-Year Comparative Study}},
author = {Y.-w. Chen and V. Bhunyanaphakul and P. Kosago and M. Finkelman and D. J. Lee},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.19.26365937},
url = {https://www.medrxiv.org/content/10.64898/2026.10.19.26365937v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Performance of Contemporary Large Language Models on the National Prosthodontic Resident Examination: A Multi-Year Comparative Study
AU - Y.-w. Chen
AU - V. Bhunyanaphakul
AU - P. Kosago
AU - M. Finkelman
AU - D. J. Lee
PY - 2026
DA - 2026/10/19
PB - medRxiv
DO - 10.64898/2026.10.19.26365937
UR - https://www.medrxiv.org/content/10.64898/2026.10.19.26365937v1
AB - PurposeTo compare the performance of three large language models (LLMs)--ChatGPT-o1, Grok 3, and Gemini 2.0 Pro--on the American College of Prosthodontists National Prosthodontic Resident Examination (NPRE). Materials and MethodsOfficial NPRE examinations from years 2015-2024 were retrieved from the American College of Prosthodontists website and used to prompt each LLM. Paid access tiers and advanced reasoning were enabled. Responses from LLMs were recorded and compared against the official answer keys for accuracy, variability across years, and consistency of responses. Duplicate questions were retained for the primary analysis and removed in a sensitivity analysis. Accuracy (percent correct) was compared using generalized estimating equations binary logistic regression (primary); Cochrans Q with Bonferroni-adjusted McNemar tests (sensitivity). Year-to-year variability (SD of annual accuracy) was compared using weighted Levene tests. Consistency across duplicate items was quantified as the percentage of perfect matches and compared using Cochrans Q with Bonferroni-adjusted McNemar tests. ResultsAcross 1,487 questions, overall accuracy differed among models (global P < 0.001). ChatGPT-o1 achieved 78.4% correct, Grok 3 76.4%, and Gemini 2.0 Pro 73.2%; ChatGPT-o1 and Grok 3s accuracy significantly exceeded Gemini 2.0 Pro (both P [≤] 0.016), with no significant difference between ChatGPT-o1 and Grok 3 (P = 0.099). After removing duplicates (n = 1,200), results were similar: ChatGPT-o1, Grok 3, and Gemini 2.0 Pro were 77.1%, 76.6%, and 71.5% correct, respectively; ChatGPT-o1 and Grok 3s accuracy was significantly higher than Gemini 2.0 Pro (both P < 0.001). Variability across years did not differ significantly (P [≥] 0.097). Grok 3 showed significantly higher perfect-match consistency (79.2%) than ChatGPT-o1 (69.4%) and Gemini 2.0 Pro (64.5%) (P [≤] 0.010). ConclusionsContemporary LLMs showed 70-80% accuracy on postgraduate prosthodontic multiple-choice content. While promising for educational support, residual error and inconsistency warrant human oversight and transparent methods reporting before curricular adoption.
ER -