Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
- 1Faculty of Dentistry, Department of Oral and Maxillofacial Surgery, Gazi University, Bişkek Cad.1.Sk. No : 4 Emek, Çankaya, Ankara, Turkey. [email protected].
- 2Faculty of Dentistry, Department of Pediatric Dentistry, Gazi University, Ankara, Turkey.
- 3Faculty of Dentistry, Department of Oral and Maxillofacial Surgery, Gazi University, Bişkek Cad.1.Sk. No : 4 Emek, Çankaya, Ankara, Turkey.
Abstract
Background
Medication-related osteonecrosis of the jaw (MRONJ) is a clinically significant complication associated with antiresorptive and antiangiogenic therapies, and its prevention and management depend on reliable, guideline-concordant information. Large language models (LLMs) are increasingly used to generate medical information, but their performance in MRONJ remains insufficiently investigated. This study compared the responses generated by ChatGPT- 5.4 Thinking, Gemini 3 Deep Think, and DeepSeek-V3.2 in thinking mode to MRONJ-related open-ended questions in terms of clinical quality, readability, and response time.Methods
In this cross-sectional comparative study, a standardized 30-item question set was developed from the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update. Each question was entered separately into each model under standardized conditions, yielding 90 responses. Outputs were anonymized and independently evaluated by three blinded oral and maxillofacial surgeons using a modified Global Quality Scale on a five-point Likert scale. A post hoc Clinical Risk Classification was additionally performed to distinguish minor informational deficiencies from omissions or inaccuracies with greater potential clinical consequences. Readability was assessed using five established English-language indices, and response time was measured with an online digital stopwatch. Appropriate statistical tests were applied, and inter-evaluator reliability was assessed using the intraclass correlation coefficient.Results
Inter-evaluator agreement was high across all models (all p < 0.001). Mean expert quality scores differed significantly among models (p < 0.001). Post hoc comparisons showed that ChatGPT-5.4 Thinking received significantly lower expert ratings than both DeepSeek-V3.2 and Gemini 3 Deep Think, whereas no statistically significant difference was detected between DeepSeek-V3.2 and Gemini 3 Deep Think. Response time also differed significantly (p < 0.001): DeepSeek-V3.2 produced the fastest responses, Gemini 3 Deep Think showed intermediate latency, and ChatGPT-5.4 Thinking had the longest response times. Readability analyses revealed significant inter-model differences across all indices, with Gemini 3 Deep Think generating the most complex outputs and DeepSeek-V3.2 producing comparatively more readable responses. Clinical Risk Classification showed that most responses were categorized as having no clinically relevant risk or low clinical risk, while moderate-risk classifications were uncommon and no high-risk responses were identified.Conclusions
Under single-prompt, single-session benchmark conditions, potentially clinically relevant differences were observed among the models in MRONJ-related response quality, readability, response time, and clinical risk classification. DeepSeek-V3.2 and Gemini 3 Deep Think achieved higher expert-rated quality scores than ChatGPT-5.4 Thinking, while DeepSeek-V3.2 also showed advantages in speed and readability. These findings do not indicate general model superiority and support using LLMs only as adjunctive informational tools, not substitutes for specialist and guideline-based clinical decision-making.
Related
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Quality, transparency, and readability of artificial intelligence chatbot responses to fibrous dysplasia-related searches: a cross-sectional comparative study
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
- Effect of guideline-assisted prompting on large language model performance in medication-related osteonecrosis of the jaw–related endodontic decision-making
BibTeX
@article{ulubas2026comparison,
title = {{Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.}},
author = {Emre Ulubaş and Damla Maide Davutluoğlu and İnci Rana Karaca and Mehmet Emin Toprak},
journal = {BMC oral health},
year = {2026},
month = jul,
doi = {10.1186/s12903-026-09421-0},
url = {https://europepmc.org/article/MED/42902318},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
AU - Emre Ulubaş
AU - Damla Maide Davutluoğlu
AU - İnci Rana Karaca
AU - Mehmet Emin Toprak
PY - 2026
DA - 2026/07/01
JO - BMC oral health
DO - 10.1186/s12903-026-09421-0
UR - https://europepmc.org/article/MED/42902318
AB - Background Medication-related osteonecrosis of the jaw (MRONJ) is a clinically significant complication associated with antiresorptive and antiangiogenic therapies, and its prevention and management depend on reliable, guideline-concordant information. Large language models (LLMs) are increasingly used to generate medical information, but their performance in MRONJ remains insufficiently investigated. This study compared the responses generated by ChatGPT- 5.4 Thinking, Gemini 3 Deep Think, and DeepSeek-V3.2 in thinking mode to MRONJ-related open-ended questions in terms of clinical quality, readability, and response time.Methods In this cross-sectional comparative study, a standardized 30-item question set was developed from the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update. Each question was entered separately into each model under standardized conditions, yielding 90 responses. Outputs were anonymized and independently evaluated by three blinded oral and maxillofacial surgeons using a modified Global Quality Scale on a five-point Likert scale. A post hoc Clinical Risk Classification was additionally performed to distinguish minor informational deficiencies from omissions or inaccuracies with greater potential clinical consequences. Readability was assessed using five established English-language indices, and response time was measured with an online digital stopwatch. Appropriate statistical tests were applied, and inter-evaluator reliability was assessed using the intraclass correlation coefficient.Results Inter-evaluator agreement was high across all models (all p < 0.001). Mean expert quality scores differed significantly among models (p < 0.001). Post hoc comparisons showed that ChatGPT-5.4 Thinking received significantly lower expert ratings than both DeepSeek-V3.2 and Gemini 3 Deep Think, whereas no statistically significant difference was detected between DeepSeek-V3.2 and Gemini 3 Deep Think. Response time also differed significantly (p < 0.001): DeepSeek-V3.2 produced the fastest responses, Gemini 3 Deep Think showed intermediate latency, and ChatGPT-5.4 Thinking had the longest response times. Readability analyses revealed significant inter-model differences across all indices, with Gemini 3 Deep Think generating the most complex outputs and DeepSeek-V3.2 producing comparatively more readable responses. Clinical Risk Classification showed that most responses were categorized as having no clinically relevant risk or low clinical risk, while moderate-risk classifications were uncommon and no high-risk responses were identified.Conclusions Under single-prompt, single-session benchmark conditions, potentially clinically relevant differences were observed among the models in MRONJ-related response quality, readability, response time, and clinical risk classification. DeepSeek-V3.2 and Gemini 3 Deep Think achieved higher expert-rated quality scores than ChatGPT-5.4 Thinking, while DeepSeek-V3.2 also showed advantages in speed and readability. These findings do not indicate general model superiority and support using LLMs only as adjunctive informational tools, not substitutes for specialist and guideline-based clinical decision-making.
SN - 1472-6831
ER -