Evaluating COPD care assistance: a comparative study of ChatGPT-3.5, ChatGPT-4.0, and ERNIE bot performance.
9 affiliations
- 1Department of Infectious Diseases, The Second Xiangya Hospital, Changsha, China.
- 2FuRong Laboratory, Changsha, China.
- 3Clinical Research Centre For Viral Hepatitis In Hunan Province, Hunan, China.
- 4Joint first authorship.
- 5Department of Respiratory and Critical Care Medicine, The Second Xiangya Hospital, Changsha, China.
- 6Research Unit of Respiratory Disease, Central South University, Changsha, China.
- 7Clinical Medical Research Center for Respiratory and Critical Care Medicine in Hunan Province, Changsha, China.
- 8Diagnosis and Treatment Center of Respiratory Disease, Central South University, Changsha, China.
- 9Department of Radiology, Changde Hospital, Xiangya School of Medicine, Central South University (The First People's Hospital of Changde City), Changde, China.
Abstract
Background
Large language model (LLM) chatbots are increasingly used to provide health information, yet their reliability in addressing chronic obstructive pulmonary disease (COPD) care questions in the Chinese context remains unclear. To evaluate and compare the performance of three publicly available LLM chatbots - ChatGPT-3.5, ChatGPT-4.0 (OpenAI), and ERNIE Bot (Baidu) - in answering chronic obstructive pulmonary disease (COPD) care questions in China.Methods
We performed a cross-sectional comparative evaluation of three publicly available LLM chatbots (ChatGPT-3.5, ChatGPT-4.0, and ERNIE Bot). We developed 40 COPD care questions spanning 7 prespecified domains relevant to patient education and self-management. All questions were submitted in Chinese in independent new chat sessions to minimise context carryover. Three consultant-level respiratory physicians independently rated each response for accuracy (1 = poor, 2 = borderline, 3 = good). We summarised accuracy as the sum of three ratings (range 3-9) and derived a majority-vote accuracy category. For responses rated 'good' by majority vote, physicians additionally rated comprehensiveness on a five-point scale. Information quality was further assessed using a modified DISCERN framework and the Patient Education Materials Assessment Tool (PEMAT) understandability and actionability domains.Results
Most responses were rated as 'good' for accuracy across chatbots. ERNIE Bot achieved higher total accuracy scores than ChatGPT-3.5 and ChatGPT-4.0, and a higher proportion of 'good' ratings in this Chinese-language evaluation. However, comprehensiveness scores for 'good' responses were low across all systems (mean ~ 1.4-2.0/5), and PEMAT actionability was limited, indicating that few responses included concrete, patient-executable steps.Conclusions
Our findings suggest that LLM chatbots may support basic COPD patient education in Chinese settings, but current responses often lack sufficient comprehensiveness and actionable guidance. These tools should not be used as standalone clinical advice and require appropriate safeguards and clinician oversight.
Related
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Evaluating the Performance of Five Leading Large Language Models in Generating Health Education Materials for Patients With Infertility: Comparative Analysis.
- Comparative evaluation of ChatGPT and Gemini feedback in scenario-based clinical reasoning among medical interns.
- Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
- Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
BibTeX
@article{dai2026evaluating,
title = {{Evaluating COPD care assistance: a comparative study of ChatGPT-3.5, ChatGPT-4.0, and ERNIE bot performance.}},
author = {Zhongshang Dai and Zihang Zeng and Zijie Zhan and Yan Chen},
journal = {Journal of global health},
year = {2026},
month = oct,
doi = {10.7189/jogh.16.04297},
url = {https://europepmc.org/article/MED/42895273},
note = {CC BY},
}
RIS
TY - JOUR
TI - Evaluating COPD care assistance: a comparative study of ChatGPT-3.5, ChatGPT-4.0, and ERNIE bot performance.
AU - Zhongshang Dai
AU - Zihang Zeng
AU - Zijie Zhan
AU - Yan Chen
PY - 2026
DA - 2026/10/19
JO - Journal of global health
DO - 10.7189/jogh.16.04297
UR - https://europepmc.org/article/MED/42895273
AB - Background Large language model (LLM) chatbots are increasingly used to provide health information, yet their reliability in addressing chronic obstructive pulmonary disease (COPD) care questions in the Chinese context remains unclear. To evaluate and compare the performance of three publicly available LLM chatbots - ChatGPT-3.5, ChatGPT-4.0 (OpenAI), and ERNIE Bot (Baidu) - in answering chronic obstructive pulmonary disease (COPD) care questions in China.Methods We performed a cross-sectional comparative evaluation of three publicly available LLM chatbots (ChatGPT-3.5, ChatGPT-4.0, and ERNIE Bot). We developed 40 COPD care questions spanning 7 prespecified domains relevant to patient education and self-management. All questions were submitted in Chinese in independent new chat sessions to minimise context carryover. Three consultant-level respiratory physicians independently rated each response for accuracy (1 = poor, 2 = borderline, 3 = good). We summarised accuracy as the sum of three ratings (range 3-9) and derived a majority-vote accuracy category. For responses rated 'good' by majority vote, physicians additionally rated comprehensiveness on a five-point scale. Information quality was further assessed using a modified DISCERN framework and the Patient Education Materials Assessment Tool (PEMAT) understandability and actionability domains.Results Most responses were rated as 'good' for accuracy across chatbots. ERNIE Bot achieved higher total accuracy scores than ChatGPT-3.5 and ChatGPT-4.0, and a higher proportion of 'good' ratings in this Chinese-language evaluation. However, comprehensiveness scores for 'good' responses were low across all systems (mean ~ 1.4-2.0/5), and PEMAT actionability was limited, indicating that few responses included concrete, patient-executable steps.Conclusions Our findings suggest that LLM chatbots may support basic COPD patient education in Chinese settings, but current responses often lack sufficient comprehensiveness and actionable guidance. These tools should not be used as standalone clinical advice and require appropriate safeguards and clinician oversight.
SN - 2047-2978
SN - 2047-2986
ER -