Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
- 1Faculty of Dentistry, Department of Periodontology, Hacettepe University
Abstract
Background There is increasing interest and ongoing debate about the integration of artificial intelligence and the large language models it supports in health science education. The performance of these models may vary depending on educational domains, clinical branches, complexity of questions, and question formats. This study aimed to compare the performance of ChatGPT-5, Gemini 2.5 Flash, and Copilot on periodontology questions and to determine whether their performance was affected by question type and topic. Methods One hundred thirty-four multiple-choice periodontology questions from the dental specialty exam in Türkiye were fed to these models. The responses were evaluated based on both their accuracy and the quality of their explanations. A Likert scale was used to examine the quality of explanation. The answers from the models were analyzed comparatively based on question type and topic. Regression analyses were performed to determine the effect of question type and topic. Results The total accuracy rates of ChatGPT-5, Gemini 2.5 Flash, and Copilot were similar. Overall, Gemini 2.5 Flash outperformed the others in terms of adequacy of response explanation (p < 0.05). Gemini provided more consistent explanations than others on basic science and clinical periodontology questions, as well as standardized and information-based questions (p < 0.05, for all). Regression analysis showed that question type and topic were related to explanation quality. Conclusions According to the findings of this study, question types and topics seem to affect the consistency and quality of large language models’ explanations more than the accuracy of the answers. This suggests that large language models are promising for periodontology education, but they still have limitations that require improvement. However, these findings should be interpreted cautiously within the limitations of the study.
Related
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Performance of large language models in answering orthodontic questions: A scoping review.
- Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non‐English Languages
- Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
BibTeX
@article{parlak2026question,
title = {{Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study}},
author = {Hanife Merva Parlak},
journal = {BMC Medical Education},
year = {2026},
month = oct,
doi = {10.1186/s12909-026-10516-z},
url = {https://doi.org/10.1186/s12909-026-10516-z},
note = {CC BY},
}
RIS
TY - JOUR
TI - Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
AU - Hanife Merva Parlak
PY - 2026
DA - 2026/10/25
JO - BMC Medical Education
DO - 10.1186/s12909-026-10516-z
UR - https://doi.org/10.1186/s12909-026-10516-z
AB - Background There is increasing interest and ongoing debate about the integration of artificial intelligence and the large language models it supports in health science education. The performance of these models may vary depending on educational domains, clinical branches, complexity of questions, and question formats. This study aimed to compare the performance of ChatGPT-5, Gemini 2.5 Flash, and Copilot on periodontology questions and to determine whether their performance was affected by question type and topic. Methods One hundred thirty-four multiple-choice periodontology questions from the dental specialty exam in Türkiye were fed to these models. The responses were evaluated based on both their accuracy and the quality of their explanations. A Likert scale was used to examine the quality of explanation. The answers from the models were analyzed comparatively based on question type and topic. Regression analyses were performed to determine the effect of question type and topic. Results The total accuracy rates of ChatGPT-5, Gemini 2.5 Flash, and Copilot were similar. Overall, Gemini 2.5 Flash outperformed the others in terms of adequacy of response explanation (p < 0.05). Gemini provided more consistent explanations than others on basic science and clinical periodontology questions, as well as standardized and information-based questions (p < 0.05, for all). Regression analysis showed that question type and topic were related to explanation quality. Conclusions According to the findings of this study, question types and topics seem to affect the consistency and quality of large language models’ explanations more than the accuracy of the answers. This suggests that large language models are promising for periodontology education, but they still have limitations that require improvement. However, these findings should be interpreted cautiously within the limitations of the study.
SN - 1472-6920
ER -