Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
- 1The Second Clinical Medical College, Guizhou University of Traditional Chinese Medicine, Guiyang, Guizhou, China
- 2Department of Pediatrics, The Second Affiliated Hospital of Guizhou University of Traditional Chinese Medicine, Guiyang, Guizhou, China
Abstract
Background: Large language model (LLM) chatbot platforms are increasingly used for health information, but their performance for pediatric immunoglobulin A vasculitis (IgAV) patient education is unclear.
Objective: To compare ChatGPT, DeepSeek, and Doubao across content completeness, quality and reliability, readability, understandability, and actionability.
Methods: Thirty-two standardized English questions were submitted to ChatGPT, DeepSeek, and Doubao on August 9, 2026, using a new conversation for each question and retaining the first complete response (96 responses). Two reviewers independently performed the original assessment, with third-reviewer adjudication. GQS, mDISCERN, QAMAI, six readability metrics, PEMAT-P, and ICC(2,1) were used. During revision, a question-specific clinical checklist was established and applied to all archived answers. The research team completed the revision scoring and confirmed the final omission results.
Results: Clinically relevant information omissions occurred in 16/96 responses (16.67%): 0/32 for ChatGPT, 5/32 for DeepSeek, and 11/32 for Doubao. Retained ratings identified outdated medical advice in 2/96 responses (2.08%) and no real-time information errors or hallucinations. At least one recorded content deficiency occurred in 18/96 responses (18.75%). Original ICC(2,1) values ranged from 0.767 to 0.954. Overall differences were significant for GQS, mDISCERN, QAMAI, PEMAT-U, and PEMAT-A (all P < .01), but not for readability metrics. Bonferroni-adjusted contrasts showed lower ARI, FKGL, CLI, and SMOG scores for DeepSeek than ChatGPT. DeepSeek had the highest PEMAT-U score; ChatGPT had the highest PEMAT-A score.
Conclusions: Performance varied by evaluation domain, and clinically relevant omissions remained a safety concern. The omission findings reflect criteria established during revision and do not establish a validated platform safety ranking. LLM platforms may supplement patient education but should not replace professional medical review.
Related
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation
- Performance of large language models in postoperative hip fracture rehabilitation counseling for older adults: a comparative evaluation of safety, accuracy, reliability, readability, and empathy
- Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study
BibTeX
@article{fei2026comparative,
title = {{Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability}},
author = {Ximing Fei and Jiao Zhang and Qiuye Li and Jianhui Yang},
journal = {Frontiers in Medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1974214},
url = {https://doi.org/10.3389/fmed.2026.1974214},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
AU - Ximing Fei
AU - Jiao Zhang
AU - Qiuye Li
AU - Jianhui Yang
PY - 2026
DA - 2026/10/21
JO - Frontiers in Medicine
DO - 10.3389/fmed.2026.1974214
UR - https://doi.org/10.3389/fmed.2026.1974214
AB - Background: Large language model (LLM) chatbot platforms are increasingly used for health information, but their performance for pediatric immunoglobulin A vasculitis (IgAV) patient education is unclear. Objective: To compare ChatGPT, DeepSeek, and Doubao across content completeness, quality and reliability, readability, understandability, and actionability. Methods: Thirty-two standardized English questions were submitted to ChatGPT, DeepSeek, and Doubao on August 9, 2026, using a new conversation for each question and retaining the first complete response (96 responses). Two reviewers independently performed the original assessment, with third-reviewer adjudication. GQS, mDISCERN, QAMAI, six readability metrics, PEMAT-P, and ICC(2,1) were used. During revision, a question-specific clinical checklist was established and applied to all archived answers. The research team completed the revision scoring and confirmed the final omission results. Results: Clinically relevant information omissions occurred in 16/96 responses (16.67%): 0/32 for ChatGPT, 5/32 for DeepSeek, and 11/32 for Doubao. Retained ratings identified outdated medical advice in 2/96 responses (2.08%) and no real-time information errors or hallucinations. At least one recorded content deficiency occurred in 18/96 responses (18.75%). Original ICC(2,1) values ranged from 0.767 to 0.954. Overall differences were significant for GQS, mDISCERN, QAMAI, PEMAT-U, and PEMAT-A (all P < .01), but not for readability metrics. Bonferroni-adjusted contrasts showed lower ARI, FKGL, CLI, and SMOG scores for DeepSeek than ChatGPT. DeepSeek had the highest PEMAT-U score; ChatGPT had the highest PEMAT-A score. Conclusions: Performance varied by evaluation domain, and clinically relevant omissions remained a safety concern. The omission findings reflect criteria established during revision and do not establish a validated platform safety ranking. LLM platforms may supplement patient education but should not replace professional medical review.
SN - 2296-858X
ER -