Benchmarking large language models on Turkish dental specialty examination questions: effects of model, discipline, and question format.
- 1Department of Endodontics, Faculty of Dentistry, Akdeniz University, Antalya, Türkiye. [email protected].
- 2Department of Endodontics, Faculty of Dentistry, Akdeniz University, Antalya, Türkiye.
Abstract
Background
Large language models (LLMs) are increasingly used as study aids in dental education, yet their performance may vary by model, discipline, and question format. This study benchmarked LLM-based systems using Turkish Dentistry Specialization Examination (DUS) Clinical Sciences questions.Methods
A public dataset of DUS Clinical Sciences multiple-choice questions from 2012 to 2021 (n = 1,027; excluding officially canceled items) was compiled from the national examination authority. Each model was evaluated once per question: ChatGPT-4o, ChatGPT-o4-mini-high, ChatGPT-5 Thinking, Gemini-2.5 Pro, Gemini-2.0 Flash, Claude Sonnet 4, Copilot, and DeepSeek-v3.1. Accuracy was scored against the official answer key. Questions were categorized as standard, K-type, or visual. Paired question-level analyses and 95% confidence intervals were used for accuracy outcomes; response time was recorded on the same endodontics question set.Results
Overall accuracy exceeded 80% for all models, and paired analysis confirmed significant differences among models (Cochran's Q(7) = 247.27; p < 0.001). ChatGPT-5 Thinking achieved the highest accuracy (94.26%), followed by Gemini-2.5 Pro (92.50%); DeepSeek-v3.1 had the lowest accuracy (80.92%). Accuracy differed markedly by question format (chi-square(2) = 391.690; p < 0.001): 89.61% for standard, 75.71% for K-type, and 55.56% for visual items. Response times differed across models (Kruskal-Wallis H(7) = 649.243; p < 0.001): Claude Sonnet 4 and Copilot were fastest, while Gemini-2.5 Pro was slowest.Conclusions
Publicly available LLM-based systems can perform strongly on DUS-level questions, but performance is sensitive to model choice, discipline, question format, and time cost. Outputs, especially for visual and K-type items, should be used as supportive educational material and verified with reliable sources.
Related
- How reliable are large language models in pediatric dentistry? Performance of seven artificial intelligence models on dental specialty examination questions (2012–2024)
- AI-augmented versus expert-authored multiple-choice questions: a psychometric comparison in a high-stakes specialty examination
- Benchmarking ten frontier large language models on 1,477 board style multiple choice questions in hematology
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study.
BibTeX
@article{sahin2026benchmarking,
title = {{Benchmarking large language models on Turkish dental specialty examination questions: effects of model, discipline, and question format.}},
author = {Nisa Özge Şahin and Alper Kuştarcı},
journal = {BMC oral health},
year = {2026},
month = aug,
doi = {10.1186/s12903-026-09323-1},
url = {https://europepmc.org/article/MED/43009941},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Benchmarking large language models on Turkish dental specialty examination questions: effects of model, discipline, and question format.
AU - Nisa Özge Şahin
AU - Alper Kuştarcı
PY - 2026
DA - 2026/08/12
JO - BMC oral health
DO - 10.1186/s12903-026-09323-1
UR - https://europepmc.org/article/MED/43009941
AB - Background Large language models (LLMs) are increasingly used as study aids in dental education, yet their performance may vary by model, discipline, and question format. This study benchmarked LLM-based systems using Turkish Dentistry Specialization Examination (DUS) Clinical Sciences questions.Methods A public dataset of DUS Clinical Sciences multiple-choice questions from 2012 to 2021 (n = 1,027; excluding officially canceled items) was compiled from the national examination authority. Each model was evaluated once per question: ChatGPT-4o, ChatGPT-o4-mini-high, ChatGPT-5 Thinking, Gemini-2.5 Pro, Gemini-2.0 Flash, Claude Sonnet 4, Copilot, and DeepSeek-v3.1. Accuracy was scored against the official answer key. Questions were categorized as standard, K-type, or visual. Paired question-level analyses and 95% confidence intervals were used for accuracy outcomes; response time was recorded on the same endodontics question set.Results Overall accuracy exceeded 80% for all models, and paired analysis confirmed significant differences among models (Cochran's Q(7) = 247.27; p < 0.001). ChatGPT-5 Thinking achieved the highest accuracy (94.26%), followed by Gemini-2.5 Pro (92.50%); DeepSeek-v3.1 had the lowest accuracy (80.92%). Accuracy differed markedly by question format (chi-square(2) = 391.690; p < 0.001): 89.61% for standard, 75.71% for K-type, and 55.56% for visual items. Response times differed across models (Kruskal-Wallis H(7) = 649.243; p < 0.001): Claude Sonnet 4 and Copilot were fastest, while Gemini-2.5 Pro was slowest.Conclusions Publicly available LLM-based systems can perform strongly on DUS-level questions, but performance is sensitive to model choice, discipline, question format, and time cost. Outputs, especially for visual and K-type items, should be used as supportive educational material and verified with reliable sources.
SN - 1472-6831
ER -