Four large language models in parent-oriented education for pediatric flexible flatfoot: Quality, readability, hallucination, and reproducibility
4 affiliations
- 1Department of Orthopaedics and Traumatology, Faculty of Medicine, Gaziantep University, Gaziantep, Türkiye.
- 2Department of Orthopaedics and Traumatology, Faculty of Medicine, Gaziantep Islamic Science and Technology University, Gaziantep, Türkiye.
- 3Department of Orthopaedics and Traumatology, Ministry of Health Aksaray Education and Research Hospital, Aksaray, Türkiye.
- 4Department of Orthopaedics and Traumatology, Ministry of Health Pazarcık State Hospital, Kahramanmaraş, Türkiye.
Abstract
Purpose
Parents increasingly consult large language model (LLM) chatbots about pediatric flexible flatfoot, a common condition that rarely requires treatment and in which the clinical task is chiefly communication. We compared four LLMs on the quality, readability, and reproducibility of parent-oriented answers.Methods
This was a cross-sectional, comparative content analysis of AI-generated text. No patients, clinical records, or human participants were involved; the units of analysis were parent-oriented questions and the chatbot responses they generated. Twenty parent-oriented questions were submitted in triplicate to ChatGPT, Claude, DeepSeek, and Gemini in independent, memory-disabled sessions (240 responses). Two blinded pediatric orthopedic surgeons scored responses (Mika grade, four-point Likert, and a hallucination taxonomy; PEMAT-P, DISCERN, and QUEST on pooled material). Readability used five indices. Between-model differences were tested with Friedman and Dunn-Bonferroni post-hoc tests; agreement and reproducibility used intraclass correlation coefficients (ICC).Results
All five readability indices differed between models (P < 0.001); none reached the recommended sixth-grade level. DeepSeek was most accessible (FRES 66.3) and ChatGPT least (FRES 17.7). On pooled-material instruments (inter-rater ICC 0.63-0.74), Claude scored highest for understandability (91.6%) and actionability (88.3%). Per-response clinical-quality measures were closely clustered, showed poor inter-rater agreement (ICC -0.01 to +0.07), and did not discriminate. Readability reproducibility across identical queries was moderate (FRES ICC 0.49-0.72).Conclusions
Clinical-quality scales did not reliably distinguish the four LLMs, whereas readability differed substantially; none met recommended readability standards, and output was only moderately reproducible. Model choice should be guided by readability and actionability, under clinician oversight.Significance of study
For a condition managed mainly through parent education, LLM selection should rest on readability and actionability rather than presumed clinical accuracy, with output used under clinician oversight.Level of evidence
V (cross-sectional comparative content analysis of AI-generated text; no human participants).
Related
- Readability of Retina Patient Education Materials Generated With a Large Language Model
- Performance of five large language models in frozen shoulder patient education: a multidimensional evaluation of readability, quality, clinical alignment, and safety
- Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Patient-facing large language model assistants for chronic constipation education: A comparative study of expert evaluation, readability, and patient acceptability
BibTeX
@article{servet2026four,
title = {{Four large language models in parent-oriented education for pediatric flexible flatfoot: Quality, readability, hallucination, and reproducibility}},
author = {Erkan Servet and Beytullah Unat and Musa Alperen Bilgin and Çağrı Karabulut and Nevzat Gönder},
journal = {Journal of children's orthopaedics},
year = {2026},
month = oct,
doi = {10.1177/18632521261500230},
url = {https://europepmc.org/article/PMC/PMC13716893},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - Four large language models in parent-oriented education for pediatric flexible flatfoot: Quality, readability, hallucination, and reproducibility
AU - Erkan Servet
AU - Beytullah Unat
AU - Musa Alperen Bilgin
AU - Çağrı Karabulut
AU - Nevzat Gönder
PY - 2026
DA - 2026/10/28
JO - Journal of children's orthopaedics
DO - 10.1177/18632521261500230
UR - https://europepmc.org/article/PMC/PMC13716893
AB - Purpose Parents increasingly consult large language model (LLM) chatbots about pediatric flexible flatfoot, a common condition that rarely requires treatment and in which the clinical task is chiefly communication. We compared four LLMs on the quality, readability, and reproducibility of parent-oriented answers.Methods This was a cross-sectional, comparative content analysis of AI-generated text. No patients, clinical records, or human participants were involved; the units of analysis were parent-oriented questions and the chatbot responses they generated. Twenty parent-oriented questions were submitted in triplicate to ChatGPT, Claude, DeepSeek, and Gemini in independent, memory-disabled sessions (240 responses). Two blinded pediatric orthopedic surgeons scored responses (Mika grade, four-point Likert, and a hallucination taxonomy; PEMAT-P, DISCERN, and QUEST on pooled material). Readability used five indices. Between-model differences were tested with Friedman and Dunn-Bonferroni post-hoc tests; agreement and reproducibility used intraclass correlation coefficients (ICC).Results All five readability indices differed between models (P < 0.001); none reached the recommended sixth-grade level. DeepSeek was most accessible (FRES 66.3) and ChatGPT least (FRES 17.7). On pooled-material instruments (inter-rater ICC 0.63-0.74), Claude scored highest for understandability (91.6%) and actionability (88.3%). Per-response clinical-quality measures were closely clustered, showed poor inter-rater agreement (ICC -0.01 to +0.07), and did not discriminate. Readability reproducibility across identical queries was moderate (FRES ICC 0.49-0.72).Conclusions Clinical-quality scales did not reliably distinguish the four LLMs, whereas readability differed substantially; none met recommended readability standards, and output was only moderately reproducible. Model choice should be guided by readability and actionability, under clinician oversight.Significance of study For a condition managed mainly through parent education, LLM selection should rest on readability and actionability rather than presumed clinical accuracy, with output used under clinician oversight.Level of evidence V (cross-sectional comparative content analysis of AI-generated text; no human participants).
SN - 1863-2521
SN - 1863-2548
ER -