Comprehensive evaluation of large language models for pediatric pectus excavatum education
Abstract
BackgroundLarge language models (LLMs) are increasingly used by patients and caregivers to find health information. However, their performance in providing Simplified Chinese education on pediatric pectus excavatum (PE) remains uncertain. This study comprehensively evaluated the responses generated by five mainstream LLMs.MethodsThirty PE-related frequently asked questions (FAQs) spanning five clinical domains were collected and submitted in Simplified Chinese to ChatGPT, Claude, DeepSeek, Qwen, and Kimi. Responses were evaluated by pediatric thoracic surgeons from three institutions for accuracy, completeness, and relevance, using a 6-point Likert scale. Readability was assessed using AlphaReadabilityChinese (ARC) 2.0 and the Chinese Readability Index Explorer (CRIE) 3.0. Understandability, actionability, and overall quality were evaluated with the Chinese Patient Education Materials Assessment Tool for Printable Materials (C-PEMAT-P) and the Global Quality Score (GQS). Friedman tests compared the models, followed by Holm-adjusted Wilcoxon signed-rank tests.ResultsAll models achieved mean accuracy, completeness, and relevance scores above 4.9 on the 6-point scale, but the leading model differed by dimension. Claude had the highest mean accuracy (5.39 ± 0.28), completeness (5.22 ± 0.22), and GQS (4.80 ± 0.41), while DeepSeek had the highest relevance (5.42 ± 0.27). Qwen was the least readable: its median CRIE grade was 7.0, compared with 5.5 to 6.0 for the other models. Only 23.3% of its responses were at Grades 1–6, compared with 66.7–86.7% for the others. Overall C-PEMAT-P scores also differed (p = 0.003), with Kimi showing the highest numerical mean total (12.17 ± 0.75). Qwen combined the highest actionability (2.57 ± 0.50) with the lowest understandability (9.47 ± 0.57) and GQS (3.10 ± 0.71).ConclusionLeading LLMs generally provided accurate, complete, and relevant Simplified Chinese answers to common pediatric PE questions, but no model performed best on every dimensions. LLM outputs should therefore be regarded as a supervised educational aid that require expert verification and adaptation to the health literacy of the intended audience, rather than as substitutes for specialist counseling.
Related
- Development and Preliminary Psychometric Evaluation of a Questionnaire on Comprehensive Protection, Quality, and Humanization in Pediatric Hospital Care
- Psychometric Evaluation of the Parental Education in Physiotherapy Scale: A Brief Measure of Family‐Centered Educational Practices in Pediatric Physiotherapy
- PsyEval: a comprehensive large language model evaluation benchmark for mental health
- Four large language models in parent-oriented education for pediatric flexible flatfoot: Quality, readability, hallucination, and reproducibility
- Evaluation of the performance and temporal variability of large language models in patient education regarding pneumothorax: a seven-day analysis
BibTeX
@article{shi2026comprehensive,
title = {{Comprehensive evaluation of large language models for pediatric pectus excavatum education}},
author = {Hao Shi and Qiang Tang and Jing Xiong and Hou-Chiang Tseng and Tao-Hsing Chang and Xiaolong Chen and Linhao Peng and Haifa Hong},
journal = {Frontiers in Public Health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1990753},
url = {https://www.frontiersin.org/articles/10.3389/fpubh.2026.1990753},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Comprehensive evaluation of large language models for pediatric pectus excavatum education
AU - Hao Shi
AU - Qiang Tang
AU - Jing Xiong
AU - Hou-Chiang Tseng
AU - Tao-Hsing Chang
AU - Xiaolong Chen
AU - Linhao Peng
AU - Haifa Hong
PY - 2026
DA - 2026/10/30
JO - Frontiers in Public Health
DO - 10.3389/fpubh.2026.1990753
UR - https://www.frontiersin.org/articles/10.3389/fpubh.2026.1990753
AB - BackgroundLarge language models (LLMs) are increasingly used by patients and caregivers to find health information. However, their performance in providing Simplified Chinese education on pediatric pectus excavatum (PE) remains uncertain. This study comprehensively evaluated the responses generated by five mainstream LLMs.MethodsThirty PE-related frequently asked questions (FAQs) spanning five clinical domains were collected and submitted in Simplified Chinese to ChatGPT, Claude, DeepSeek, Qwen, and Kimi. Responses were evaluated by pediatric thoracic surgeons from three institutions for accuracy, completeness, and relevance, using a 6-point Likert scale. Readability was assessed using AlphaReadabilityChinese (ARC) 2.0 and the Chinese Readability Index Explorer (CRIE) 3.0. Understandability, actionability, and overall quality were evaluated with the Chinese Patient Education Materials Assessment Tool for Printable Materials (C-PEMAT-P) and the Global Quality Score (GQS). Friedman tests compared the models, followed by Holm-adjusted Wilcoxon signed-rank tests.ResultsAll models achieved mean accuracy, completeness, and relevance scores above 4.9 on the 6-point scale, but the leading model differed by dimension. Claude had the highest mean accuracy (5.39 ± 0.28), completeness (5.22 ± 0.22), and GQS (4.80 ± 0.41), while DeepSeek had the highest relevance (5.42 ± 0.27). Qwen was the least readable: its median CRIE grade was 7.0, compared with 5.5 to 6.0 for the other models. Only 23.3% of its responses were at Grades 1–6, compared with 66.7–86.7% for the others. Overall C-PEMAT-P scores also differed (p = 0.003), with Kimi showing the highest numerical mean total (12.17 ± 0.75). Qwen combined the highest actionability (2.57 ± 0.50) with the lowest understandability (9.47 ± 0.57) and GQS (3.10 ± 0.71).ConclusionLeading LLMs generally provided accurate, complete, and relevant Simplified Chinese answers to common pediatric PE questions, but no model performed best on every dimensions. LLM outputs should therefore be regarded as a supervised educational aid that require expert verification and adaptation to the health literacy of the intended audience, rather than as substitutes for specialist counseling.
SN - 2296-2565
ER -