Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- 1Department of Oncology, Affiliated Hospital of Nanjing University of Chinese Medicine, Nanjing, Jiangsu, China.
- 2The First Clinical Medical College of Nanjing University of Chinese Medicine, Nanjing, China.
- 3Jiangsu Collaborative Innovation Center of TCM Cancer Prevention and Treatment, Nanjing, China.
Abstract
Large language models (LLMs) are increasingly used as sources of health information. This study evaluated the quality, reliability, readability, understandability, and actionability of responses generated by three mainstream LLMs to esophageal cancer-related questions. Thirty esophageal cancer-related questions covering six domains were identified from Google Trends, clinical practice guidelines, and social media and submitted to three LLMs, yielding 90 responses. Responses were evaluated using the Global Quality Scale (GQS), modified DISCERN (mDISCERN), Ensuring Quality Information for Patients (EQIP), Quality Assessment of Medical Artificial Intelligence (QAMAI), Patient Education Materials Assessment Tool (PEMAT), and six readability indices. Differences among the models were assessed using repeated-measures ANOVA or the Friedman test according to data distribution. Crucial information missing was the most common content deficiency (8.9%). Overall GQS, mDISCERN, QAMAI, and EQIP scores were 4.00 (IQR: 4.00-4.00), 4.00 (IQR: 3.50-4.00), 22.00 (IQR: 21.50-25.00), and 71.88 (IQR: 62.50-75.00), respectively. DeepSeek performed best overall, followed by ChatGPT and Doubao. None of the three models met the recommended sixth-grade reading level, and all had PEMAT-A scores below the recommended 70% threshold. Readability was generally not significantly associated with quality or reliability, except for a few weak negative correlations. LLMs demonstrated generally favorable quality and reliability when answering esophageal cancer-related questions, but readability and actionability remained limited. They may serve as supplementary tools for patients seeking esophageal cancer-related health information but should not replace professional medical consultation.
Related
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation
- Performance of large language models in postoperative hip fracture rehabilitation counseling for older adults: a comparative evaluation of safety, accuracy, reliability, readability, and empathy
- Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study
- Multidimensional evaluation of generative AI chatbots for public consultation on sudden cardiac death: safety, accuracy, empathy, reliability, quality, and readability across six models
BibTeX
@article{hang2026comparative,
title = {{Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.}},
author = {Ying Ying Hang and Rui Yuan and Man Lu and Fang Wen Long and His Quan Hsu and Lu Bai and Jun Qian},
journal = {Frontiers in medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1973075},
url = {https://europepmc.org/article/MED/42952682},
note = {CC BY},
}
RIS
TY - JOUR
TI - Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
AU - Ying Ying Hang
AU - Rui Yuan
AU - Man Lu
AU - Fang Wen Long
AU - His Quan Hsu
AU - Lu Bai
AU - Jun Qian
PY - 2026
DA - 2026/10/14
JO - Frontiers in medicine
DO - 10.3389/fmed.2026.1973075
UR - https://europepmc.org/article/MED/42952682
AB - Large language models (LLMs) are increasingly used as sources of health information. This study evaluated the quality, reliability, readability, understandability, and actionability of responses generated by three mainstream LLMs to esophageal cancer-related questions. Thirty esophageal cancer-related questions covering six domains were identified from Google Trends, clinical practice guidelines, and social media and submitted to three LLMs, yielding 90 responses. Responses were evaluated using the Global Quality Scale (GQS), modified DISCERN (mDISCERN), Ensuring Quality Information for Patients (EQIP), Quality Assessment of Medical Artificial Intelligence (QAMAI), Patient Education Materials Assessment Tool (PEMAT), and six readability indices. Differences among the models were assessed using repeated-measures ANOVA or the Friedman test according to data distribution. Crucial information missing was the most common content deficiency (8.9%). Overall GQS, mDISCERN, QAMAI, and EQIP scores were 4.00 (IQR: 4.00-4.00), 4.00 (IQR: 3.50-4.00), 22.00 (IQR: 21.50-25.00), and 71.88 (IQR: 62.50-75.00), respectively. DeepSeek performed best overall, followed by ChatGPT and Doubao. None of the three models met the recommended sixth-grade reading level, and all had PEMAT-A scores below the recommended 70% threshold. Readability was generally not significantly associated with quality or reliability, except for a few weak negative correlations. LLMs demonstrated generally favorable quality and reliability when answering esophageal cancer-related questions, but readability and actionability remained limited. They may serve as supplementary tools for patients seeking esophageal cancer-related health information but should not replace professional medical consultation.
SN - 2296-858X
ER -