Navigating digital surgical advice: evaluating the reliability of large language models for pediatric phimosis
- 1Department of Surgery, The University of Hong Kong, Hong Kong, China
Abstract
Background: Phimosis is an extremely common pediatric surgical condition. Given the convenience and accessibility brought by artificial intelligence, the readability, quality and accuracy evaluation on the information provided by large language models (LLM) is of utmost importance in providing comprehensive clinical care.
Methods: Six case questions relating to phimosis on its definition, diagnosis, treatment options and potential complications were inputted into each LLM-Chatbots (ChatGPT-3.5, Gemini-1.5 and Claude-3). All responses were reformatted, shuffled and recombined into three passages for single-blind scoring on accuracy by three pediatric surgeons with reference to European Association of Urology (EAU) pediatric urology guidelines. The responses were also assessed for readability using the Gunning-Fog score, Simple Measure of Gobbledygook (SMOG) index and Dale-Chall score and for quality using the DISCERN score.
Results: Comprehension of the information was considered difficult with a mean Gunning-Fog score, SMOG index score and Dale-Chall score of 14.2±2.6, 11.3±2.2 and 9.9±0.9, respectively. The mean summed DISCERN score was 39.6±2 out of 80. Claude-3-sonnet was considered of poor quality while others were fair. The mean accuracy score of the LLM-chatbots was 4.4±0.4 out of 5. The three LLMs performed similarly on readability and accuracy. The LLM-chatbots generally scored well in providing treatment options and their mechanisms, while lacking references and information regarding treatment benefits, risks of alternative treatments and impact on quality of life.
Conclusions: The three LLM-chatbots showed comparable and satisfactory accuracy in providing information on pediatric phimosis; however, their readability and overall quality were suboptimal. These limitations may affect parent counseling, shared decision-making and informed consent. LLM-generated information should, therefore, supplement, not replace, clinician guidance and reliable medical resources.
Level of evidence: III.
Related
- Evaluating the reliability, quality, and readability of AI-generated patient education on hallux valgus: a comparative study of large language models
- Navigating the Unknown: A Surgical Perspective on Thoracic ALK-Positive Histiocytosis With Literature Review.
- Navigating Uncertainty: Characterizing Rebound Lactic Acidosis in Pediatric Normothermic Machine Perfusion
- Evaluating the quality and reliability of YouTube videos about sepsis
- When are large language models medical devices? Navigating the MDR and AI Act for healthcare applications.
BibTeX
@article{fung2026navigating,
title = {{Navigating digital surgical advice: evaluating the reliability of large language models for pediatric phimosis}},
author = {Adrian Chi Heng Fung and John Yeh Joon Park and Eunice Cheuk Lam Lee and Carol Wing Yan Wong and Fanny Yeung and Ivy Hau Yee Chan and Patrick Ho Yu Chung and Kenneth KY Wong},
journal = {World Journal of Pediatric Surgery},
year = {2026},
month = nov,
doi = {10.1136/wjps-2026-001276},
url = {https://doi.org/10.1136/wjps-2026-001276},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Navigating digital surgical advice: evaluating the reliability of large language models for pediatric phimosis
AU - Adrian Chi Heng Fung
AU - John Yeh Joon Park
AU - Eunice Cheuk Lam Lee
AU - Carol Wing Yan Wong
AU - Fanny Yeung
AU - Ivy Hau Yee Chan
AU - Patrick Ho Yu Chung
AU - Kenneth KY Wong
PY - 2026
DA - 2026/11/03
JO - World Journal of Pediatric Surgery
DO - 10.1136/wjps-2026-001276
UR - https://doi.org/10.1136/wjps-2026-001276
AB - Background: Phimosis is an extremely common pediatric surgical condition. Given the convenience and accessibility brought by artificial intelligence, the readability, quality and accuracy evaluation on the information provided by large language models (LLM) is of utmost importance in providing comprehensive clinical care. Methods: Six case questions relating to phimosis on its definition, diagnosis, treatment options and potential complications were inputted into each LLM-Chatbots (ChatGPT-3.5, Gemini-1.5 and Claude-3). All responses were reformatted, shuffled and recombined into three passages for single-blind scoring on accuracy by three pediatric surgeons with reference to European Association of Urology (EAU) pediatric urology guidelines. The responses were also assessed for readability using the Gunning-Fog score, Simple Measure of Gobbledygook (SMOG) index and Dale-Chall score and for quality using the DISCERN score. Results: Comprehension of the information was considered difficult with a mean Gunning-Fog score, SMOG index score and Dale-Chall score of 14.2±2.6, 11.3±2.2 and 9.9±0.9, respectively. The mean summed DISCERN score was 39.6±2 out of 80. Claude-3-sonnet was considered of poor quality while others were fair. The mean accuracy score of the LLM-chatbots was 4.4±0.4 out of 5. The three LLMs performed similarly on readability and accuracy. The LLM-chatbots generally scored well in providing treatment options and their mechanisms, while lacking references and information regarding treatment benefits, risks of alternative treatments and impact on quality of life. Conclusions: The three LLM-chatbots showed comparable and satisfactory accuracy in providing information on pediatric phimosis; however, their readability and overall quality were suboptimal. These limitations may affect parent counseling, shared decision-making and informed consent. LLM-generated information should, therefore, supplement, not replace, clinician guidance and reliable medical resources. Level of evidence: III.
SN - 2096-6938
SN - 2516-5410
ER -