Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non‐English Languages
9 affiliations
- 1David Geffen School of Medicine University of California, Los Angeles Los Angeles California USA
- 2Division of Otolaryngology Cedars‐Sinai Medical Center Los Angeles California USA
- 3Department of Otolaryngology Henry Ford Health Detroit Michigan USA
- 4Department of Otolaryngology‐Head and Neck Surgery Beijing Tongren Hospital Beijing China
- 5Department of Otolaryngology Taiwan University Hospital Taipei China
- 6Department of Otorhinolaryngology‐Head and Neck Surgery Seoul National University Hospital Seoul Republic of Korea
- 7Department of Otorhinolaryngology Seoul National University Bundang Hospital Seongnam Republic of Korea
- 8Department of Otolaryngology‐Head and Neck Surgery University of Washington Seattle Washington USA
- 9Department of Head and Neck Surgery Kaiser Permanente West Los Angeles Los Angeles California USA
Abstract
Background Large language models (LLMs) such as ChatGPT are being explored for various medical applications, but their performance across languages is not established. Despite the promising results in English from previous studies, it is unclear how ChatGPT performs in other languages. We assessed the ability of ChatGPT‐4 to answer common otolaryngology‐related frequently asked questions in Chinese, Korean, and Spanish in this study. Methods Eighteen clinical questions were selected from the American Academy of Otolaryngology‐Head and Neck Surgery Foundation guidelines and ENT Health across six subspecialties. Bilingual clinicians translated the questions to Chinese, Korean, or Spanish and input them into ChatGPT‐4 (OpenAI, San Francisco, USA). The responses were rated on a five‐point Likert scale for accuracy, completeness, and similarity to English reference answers. A linear mixed‐effects model with language as a fixed effect and random intercepts for question and rater was used to compare performance across languages. Inter‐rater reliability was assessed using intraclass correlation coefficients and Krippendorff's α. Subjective evaluations were also collected. Results ChatGPT‐4's performance differed significantly by language in all assessment domains. Responses in Spanish scored highest throughout and most frequently were characterized as clear and organized. Korean answers were generally accurate but tended to be more conservative and less detailed. Chinese answers showed greater variability, with reviewers noting a mix of clear and less accurate outputs. Conclusion The clinical performance of ChatGPT‐4 differed across Chinese, Korean, and Spanish. These findings underscore the importance of ongoing evaluation and refinement of LLMs to ensure they provide reliable multilingual support in clinical practice.
Related
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Performance of large language models in answering orthodontic questions: A scoping review.
- Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
- Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study
- Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
BibTeX
@article{oh2026performance,
title = {{Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non‐English Languages}},
author = {Eugene Oh and Arthur W. Wu and Laura Garcia‐Rodriguez and Zhen‐Xiao Huang and Wei‐Chung Hsu and Jeongkyou Kim and Young Chul Kim and Yi‐Tsen Lin and Hae Chan Park and Hector Andres Perez and Juan San Juan and Gene Liu and Matthew Lee and Dennis M. Tang},
journal = {World Journal of Otorhinolaryngology-Head and Neck Surgery},
year = {2026},
month = oct,
doi = {10.1002/wjo2.70153},
url = {https://doi.org/10.1002/wjo2.70153},
note = {CC BY},
}
RIS
TY - JOUR
TI - Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non‐English Languages
AU - Eugene Oh
AU - Arthur W. Wu
AU - Laura Garcia‐Rodriguez
AU - Zhen‐Xiao Huang
AU - Wei‐Chung Hsu
AU - Jeongkyou Kim
AU - Young Chul Kim
AU - Yi‐Tsen Lin
AU - Hae Chan Park
AU - Hector Andres Perez
AU - Juan San Juan
AU - Gene Liu
AU - Matthew Lee
AU - Dennis M. Tang
PY - 2026
DA - 2026/10/23
JO - World Journal of Otorhinolaryngology-Head and Neck Surgery
DO - 10.1002/wjo2.70153
UR - https://doi.org/10.1002/wjo2.70153
AB - Background Large language models (LLMs) such as ChatGPT are being explored for various medical applications, but their performance across languages is not established. Despite the promising results in English from previous studies, it is unclear how ChatGPT performs in other languages. We assessed the ability of ChatGPT‐4 to answer common otolaryngology‐related frequently asked questions in Chinese, Korean, and Spanish in this study. Methods Eighteen clinical questions were selected from the American Academy of Otolaryngology‐Head and Neck Surgery Foundation guidelines and ENT Health across six subspecialties. Bilingual clinicians translated the questions to Chinese, Korean, or Spanish and input them into ChatGPT‐4 (OpenAI, San Francisco, USA). The responses were rated on a five‐point Likert scale for accuracy, completeness, and similarity to English reference answers. A linear mixed‐effects model with language as a fixed effect and random intercepts for question and rater was used to compare performance across languages. Inter‐rater reliability was assessed using intraclass correlation coefficients and Krippendorff's α. Subjective evaluations were also collected. Results ChatGPT‐4's performance differed significantly by language in all assessment domains. Responses in Spanish scored highest throughout and most frequently were characterized as clear and organized. Korean answers were generally accurate but tended to be more conservative and less detailed. Chinese answers showed greater variability, with reviewers noting a mix of clear and less accurate outputs. Conclusion The clinical performance of ChatGPT‐4 differed across Chinese, Korean, and Spanish. These findings underscore the importance of ongoing evaluation and refinement of LLMs to ensure they provide reliable multilingual support in clinical practice.
SN - 2589-1081
SN - 2095-8811
ER -