Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives
- 1Affiliated Hospital of North Sichuan Medical College
Abstract
ObjectiveTo evaluate the information quality, reliability, readability, and content deficits of ChatGPT, DeepSeek, and Doubao in answering frequently asked questions about bee stings.
MethodsTwenty-five high-frequency patient questions were selected based on Google Trends, domestic health platforms, and allergy guidelines, and submitted to the three LLMs, yielding 75 responses. Two emergency physicians independently assessed quality using GQS, mDISCERN, QAMAI, EQIP, and PEMAT-U/A. Readability was evaluated using six indices (ARI, FRES, GFI, FKGL, CLI, SMOG). Content deficits were classified qualitatively. Twenty emergency department patients evaluated the comprehensibility and credibility of AI responses.
ResultsSignificant differences were found among the three models on all quality metrics (all P < 0.01). DeepSeek achieved the highest mDISCERN score (median 21.00); ChatGPT led on QAMAI (25.00) and EQIP (75.00%); Doubao ranked lowest on all quality indicators but highest on PEMAT-U (73.00%). No model met the sixth-grade reading level standard. Eleven content deficits were identified: outdated medical advice (45.45%), omissions (36.36%), and hallucinations (18.18%). Doubao received the highest patient preference (45.0%).
ConclusionThe three LLMs exhibited distinct strengths: DeepSeek in source reliability, ChatGPT in overall quality, and Doubao in understandability. All models had excessively high readability levels, and outdated medical advice was the predominant content deficit.
Related
- Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability
- AI-generated high-intensity interval training information: a cross-sectional comparative evaluation of quality and readability
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Quality, readability, and information fidelity of large language model–generated versus physician-authored informed-consent content for targeted therapies: a blinded, paired comparative study
BibTeX
@misc{xu2026quality,
title = {{Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives}},
author = {R. Xu and L. Yang and X. Zhang and M. Cang and J. Lin and X. Jiang},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.29.26364282},
url = {https://www.medrxiv.org/content/10.64898/2026.09.29.26364282v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives
AU - R. Xu
AU - L. Yang
AU - X. Zhang
AU - M. Cang
AU - J. Lin
AU - X. Jiang
PY - 2026
DA - 2026/09/29
PB - medRxiv
DO - 10.64898/2026.09.29.26364282
UR - https://www.medrxiv.org/content/10.64898/2026.09.29.26364282v1
AB - ObjectiveTo evaluate the information quality, reliability, readability, and content deficits of ChatGPT, DeepSeek, and Doubao in answering frequently asked questions about bee stings. MethodsTwenty-five high-frequency patient questions were selected based on Google Trends, domestic health platforms, and allergy guidelines, and submitted to the three LLMs, yielding 75 responses. Two emergency physicians independently assessed quality using GQS, mDISCERN, QAMAI, EQIP, and PEMAT-U/A. Readability was evaluated using six indices (ARI, FRES, GFI, FKGL, CLI, SMOG). Content deficits were classified qualitatively. Twenty emergency department patients evaluated the comprehensibility and credibility of AI responses. ResultsSignificant differences were found among the three models on all quality metrics (all P < 0.01). DeepSeek achieved the highest mDISCERN score (median 21.00); ChatGPT led on QAMAI (25.00) and EQIP (75.00%); Doubao ranked lowest on all quality indicators but highest on PEMAT-U (73.00%). No model met the sixth-grade reading level standard. Eleven content deficits were identified: outdated medical advice (45.45%), omissions (36.36%), and hallucinations (18.18%). Doubao received the highest patient preference (45.0%). ConclusionThe three LLMs exhibited distinct strengths: DeepSeek in source reliability, ChatGPT in overall quality, and Doubao in understandability. All models had excessively high readability levels, and outdated medical advice was the predominant content deficit.
ER -