Comparative evaluation of four large language models for postoperative patient education after septorhinoplasty: A bilingual cross-sectional study
- 1Department of Otorhinolaryngology- Head and Neck Surgery, Etlik City Hospital, Ankara, Türkiye
Abstract
Aim: To compare the quality, readability, and emergency referral performance of four major language models in answering post-septorhinoplasty patient questions in Turkish and English.
Materials and Methods: A cross-sectional evaluation study comprised 31 questions: 20 routine postoperative and 11 emergency/red-flag scenarios. Each question was submitted once to ChatGPT, Google Gemini, DeepSeek, and Claude in Turkish and English within independent conversation windows. Two otolaryngologists independently evaluated the responses using the Quality Analysis of Medical Artificial Intelligence tool. English responses were additionally assessed using Flesch Reading Ease, Flesch-Kincaid Grade Level, and Simple Measure of Gobbledygook indices. Emergency referral accuracy was evaluated separately.
Results: Total quality scores differed significantly among models for Turkish routine postoperative questions (p<0.001, Kendall’s W=0.775) and for Turkish emergency/red-flag questions (p=0.009, W=0.350), whereas the corresponding overall differences for English questions were not statistically significant. Within-model language comparisons showed significant Turkish-English differences for routine questions across all four models, whereas no language-related differences were observed for emergency/red-flag questions. ChatGPT, DeepSeek, and Claude provided appropriate emergency referrals in all 22 responses, whereas Gemini did so in 16 of 22 responses (72.7%). All models exceeded recommended readability thresholds for patient education materials.
Conclusion: The performance of large language models in post-septorhinoplasty patient education varied by language context and question category. Although emergency referral performance was generally high, model-specific omissions occurred, and all models exceeded recommended readability thresholds. These findings support language-specific evaluation, simplification to improve readability, and expert review before clinical use.
Related
- Comparative Evaluation of Postoperative Corneal Astigmatic Correction Following Pterygium Excision by Bare Sclera Technique Versus Conjunctival Limbal Autograft: A Prospective Observational Study
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Patient-facing large language model assistants for chronic constipation education: A comparative study of expert evaluation, readability, and patient acceptability
- Performance of large language models in postoperative hip fracture rehabilitation counseling for older adults: a comparative evaluation of safety, accuracy, reliability, readability, and empathy
- Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability
BibTeX
@article{kucuktag2026comparative,
title = {{Comparative evaluation of four large language models for postoperative patient education after septorhinoplasty: A bilingual cross-sectional study}},
author = {Zülküf Küçüktağ and Aykut Özdoğan and Gökçe Saygı Uysal and Yağmur Uluhan},
journal = {Northwestern Medical Journal},
year = {2026},
month = oct,
doi = {10.54307/nwmj.2026.262},
url = {https://doi.org/10.54307/nwmj.2026.262},
note = {CC BY},
}
RIS
TY - JOUR
TI - Comparative evaluation of four large language models for postoperative patient education after septorhinoplasty: A bilingual cross-sectional study
AU - Zülküf Küçüktağ
AU - Aykut Özdoğan
AU - Gökçe Saygı Uysal
AU - Yağmur Uluhan
PY - 2026
DA - 2026/10/26
JO - Northwestern Medical Journal
DO - 10.54307/nwmj.2026.262
UR - https://doi.org/10.54307/nwmj.2026.262
AB - Aim: To compare the quality, readability, and emergency referral performance of four major language models in answering post-septorhinoplasty patient questions in Turkish and English. Materials and Methods: A cross-sectional evaluation study comprised 31 questions: 20 routine postoperative and 11 emergency/red-flag scenarios. Each question was submitted once to ChatGPT, Google Gemini, DeepSeek, and Claude in Turkish and English within independent conversation windows. Two otolaryngologists independently evaluated the responses using the Quality Analysis of Medical Artificial Intelligence tool. English responses were additionally assessed using Flesch Reading Ease, Flesch-Kincaid Grade Level, and Simple Measure of Gobbledygook indices. Emergency referral accuracy was evaluated separately. Results: Total quality scores differed significantly among models for Turkish routine postoperative questions (p<0.001, Kendall’s W=0.775) and for Turkish emergency/red-flag questions (p=0.009, W=0.350), whereas the corresponding overall differences for English questions were not statistically significant. Within-model language comparisons showed significant Turkish-English differences for routine questions across all four models, whereas no language-related differences were observed for emergency/red-flag questions. ChatGPT, DeepSeek, and Claude provided appropriate emergency referrals in all 22 responses, whereas Gemini did so in 16 of 22 responses (72.7%). All models exceeded recommended readability thresholds for patient education materials. Conclusion: The performance of large language models in post-septorhinoplasty patient education varied by language context and question category. Although emergency referral performance was generally high, model-specific omissions occurred, and all models exceeded recommended readability thresholds. These findings support language-specific evaluation, simplification to improve readability, and expert review before clinical use.
SN - 2979-9538
ER -