Comparative evaluation of large language model chatbots for pheochromocytoma education: accuracy, information quality, transparency, and readability
- 1Department of Urology, Shenzhen Clinical Research Center for Urology and Nephrology, Peking University Shenzhen Hospital
- 2Department of Urology, South China Hospital, Medical School, Shenzhen University
Abstract
Background: Large language model (LLM)-based chatbots are increasingly used for medical information, but their suitability for pheochromocytoma education remains uncertain. We compared four contemporary LLMs for knowledge accuracy, information quality, transparency, and readability.
Methods: ChatGPT (ChatGPT-5.6 Sol), DeepSeek (DeepSeek-V4 Flash), Gemini (Gemini-3.6 Flash), and Grok (Grok-4.5 Fast) answered 200 pheochromocytoma-related multiple-choice questions across three independent trials. Google Trends and clinician input were used to develop 15 patient-centered questions. Responses were evaluated using DISCERN, EQIP, GQS, JAMA benchmark criteria, and six readability indices.
Results: Overall knowledge accuracy differed among the four models ( p = 0.0012), with mean accuracy rates of 93.00% for ChatGPT, 91.17% for DeepSeek, 90.50% for Gemini, and 81.67% for Grok. However, after adjustment for multiple comparisons, statistically significant pairwise differences were limited to ChatGPT versus Grok for overall accuracy (adjusted p = 0.0129) and clinical-presentation questions (adjusted p = 0.0088). No significant differences were identified among ChatGPT, DeepSeek, and Gemini. Exploratory assessments of 15 patient-oriented questions suggested differences in information quality, transparency, and readability across models. ChatGPT showed comparatively favorable information quality scores, whereas DeepSeek exhibited a more favorable descriptive readability profile. Nevertheless, transparency was limited across all models, and none achieved the recommended sixth-grade readability standard.
Conclusion: The evaluated LLM-based chatbots demonstrated generally favorable performance on structured pheochromocytoma-related knowledge questions, although statistically significant pairwise differences in accuracy were limited. Findings from the exploratory patient-oriented assessments suggested potential differences in information quality and readability but require confirmation in larger studies. Persistent limitations in transparency and readability highlight the importance of clinician oversight, source verification, and linguistic simplification before these tools are used for patient education.
Related
- Evaluating AI chatbots as sources of public-facing encephalitis health information: a cross-sectional comparative study of quality, transparency, and readability
- Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives
- Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability
- Quality, transparency, and readability of artificial intelligence chatbot responses to fibrous dysplasia-related searches: a cross-sectional comparative study
- AI-generated high-intensity interval training information: a cross-sectional comparative evaluation of quality and readability
BibTeX
@article{zhao2026comparative,
title = {{Comparative evaluation of large language model chatbots for pheochromocytoma education: accuracy, information quality, transparency, and readability}},
author = {Jiadong Zhao and Fuyu Guo and Yang Liu and Peng Yu and Yi Zhang and Qingshan Yang and Shuolei Sun},
journal = {Frontiers in Public Health},
year = {2026},
month = nov,
doi = {10.3389/fpubh.2026.1983835},
url = {https://www.frontiersin.org/articles/10.3389/fpubh.2026.1983835/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Comparative evaluation of large language model chatbots for pheochromocytoma education: accuracy, information quality, transparency, and readability
AU - Jiadong Zhao
AU - Fuyu Guo
AU - Yang Liu
AU - Peng Yu
AU - Yi Zhang
AU - Qingshan Yang
AU - Shuolei Sun
PY - 2026
DA - 2026/11/12
JO - Frontiers in Public Health
DO - 10.3389/fpubh.2026.1983835
UR - https://www.frontiersin.org/articles/10.3389/fpubh.2026.1983835/full
AB - Background: Large language model (LLM)-based chatbots are increasingly used for medical information, but their suitability for pheochromocytoma education remains uncertain. We compared four contemporary LLMs for knowledge accuracy, information quality, transparency, and readability. Methods: ChatGPT (ChatGPT-5.6 Sol), DeepSeek (DeepSeek-V4 Flash), Gemini (Gemini-3.6 Flash), and Grok (Grok-4.5 Fast) answered 200 pheochromocytoma-related multiple-choice questions across three independent trials. Google Trends and clinician input were used to develop 15 patient-centered questions. Responses were evaluated using DISCERN, EQIP, GQS, JAMA benchmark criteria, and six readability indices. Results: Overall knowledge accuracy differed among the four models ( p = 0.0012), with mean accuracy rates of 93.00% for ChatGPT, 91.17% for DeepSeek, 90.50% for Gemini, and 81.67% for Grok. However, after adjustment for multiple comparisons, statistically significant pairwise differences were limited to ChatGPT versus Grok for overall accuracy (adjusted p = 0.0129) and clinical-presentation questions (adjusted p = 0.0088). No significant differences were identified among ChatGPT, DeepSeek, and Gemini. Exploratory assessments of 15 patient-oriented questions suggested differences in information quality, transparency, and readability across models. ChatGPT showed comparatively favorable information quality scores, whereas DeepSeek exhibited a more favorable descriptive readability profile. Nevertheless, transparency was limited across all models, and none achieved the recommended sixth-grade readability standard. Conclusion: The evaluated LLM-based chatbots demonstrated generally favorable performance on structured pheochromocytoma-related knowledge questions, although statistically significant pairwise differences in accuracy were limited. Findings from the exploratory patient-oriented assessments suggested potential differences in information quality and readability but require confirmation in larger studies. Persistent limitations in transparency and readability highlight the importance of clinician oversight, source verification, and linguistic simplification before these tools are used for patient education.
SN - 2296-2565
ER -