Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability
- 1Department of Ophthalmology, Zhongnan Hospital of Wuhan University , , ,
Abstract
Background: Patients increasingly use large language models (LLMs) to seek medical information, but their performance in answering diabetic retinopathy (DR) questions remains insufficiently characterized.
Objective: To evaluate the safety, accuracy, patient-centered communication, information quality, transparency, and readability of diabetic retinopathy information generated by widely accessible large language models.
Methods: Fifty standardized DR patient questions were submitted to ChatGPT, Gemini, DeepSeek, Doubao, Copilot, and Perplexity. Responses were independently evaluated by five raters using guideline-based reference answer elements. Rater-based outcomes included Safety, Accuracy, Empathy, DISCERN, EQIP, JAMA benchmark criteria, and Global Quality Score (GQS). Readability was assessed using the Automated Readability Index, Coleman–Liau Index, Flesch–Kincaid Grade Level, Gunning Fog Index, SMOG, and Flesch Reading Ease Score. Inter-rater agreement and between-model differences were analyzed using appropriate agreement statistics and paired nonparametric tests.
Results: Inter-rater agreement was moderate for Safety (Fleiss’ κ = 0.581) and good for non-binary outcomes, with ICC (2,1) values ranging from 0.806 to 0.870. Most responses were classified as safe, with safe response rates ranging from 86.0 to 94.0%, and no significant between-model difference was observed. Accuracy scores were generally high across models, although between-model differences were statistically significant with a small effect size. Empathy varied more noticeably, with Copilot scoring highest and Doubao and Perplexity scoring lower. Perplexity achieved the highest DISCERN and JAMA scores, while Copilot performed best on EQIP. GQS differences were small. Readability differed significantly across models, with Gemini generally producing the most readable responses; however, all models exceeded the recommended sixth-grade reading level, and all Flesch Reading Ease Scores were below the recommended threshold.
Conclusion: No single LLM performed best across all domains. Although the evaluated models generally provided safe and accurate responses to DR patient questions, limitations remained in safety escalation, empathy, transparency, and readability. LLM-generated DR information may support general patient education but should not replace professional ophthalmic consultation.
Related
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Patient-facing large language model assistants for chronic constipation education: A comparative study of expert evaluation, readability, and patient acceptability
- Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives
- Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
- Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation
BibTeX
@article{chen2026large,
title = {{Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability}},
author = {Zhiqi Chen and Wen Liu and Jing Zhang and Tian Zheng and Ying Li and Qing Bao and Pingting Ji and Hongmei Yao and Ying He},
journal = {Frontiers in public health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1945318},
url = {https://europepmc.org/article/PMC/PMC13690771},
note = {CC BY},
}
RIS
TY - JOUR
TI - Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability
AU - Zhiqi Chen
AU - Wen Liu
AU - Jing Zhang
AU - Tian Zheng
AU - Ying Li
AU - Qing Bao
AU - Pingting Ji
AU - Hongmei Yao
AU - Ying He
PY - 2026
DA - 2026/10/09
JO - Frontiers in public health
DO - 10.3389/fpubh.2026.1945318
UR - https://europepmc.org/article/PMC/PMC13690771
AB - Background: Patients increasingly use large language models (LLMs) to seek medical information, but their performance in answering diabetic retinopathy (DR) questions remains insufficiently characterized. Objective: To evaluate the safety, accuracy, patient-centered communication, information quality, transparency, and readability of diabetic retinopathy information generated by widely accessible large language models. Methods: Fifty standardized DR patient questions were submitted to ChatGPT, Gemini, DeepSeek, Doubao, Copilot, and Perplexity. Responses were independently evaluated by five raters using guideline-based reference answer elements. Rater-based outcomes included Safety, Accuracy, Empathy, DISCERN, EQIP, JAMA benchmark criteria, and Global Quality Score (GQS). Readability was assessed using the Automated Readability Index, Coleman–Liau Index, Flesch–Kincaid Grade Level, Gunning Fog Index, SMOG, and Flesch Reading Ease Score. Inter-rater agreement and between-model differences were analyzed using appropriate agreement statistics and paired nonparametric tests. Results: Inter-rater agreement was moderate for Safety (Fleiss’ κ = 0.581) and good for non-binary outcomes, with ICC (2,1) values ranging from 0.806 to 0.870. Most responses were classified as safe, with safe response rates ranging from 86.0 to 94.0%, and no significant between-model difference was observed. Accuracy scores were generally high across models, although between-model differences were statistically significant with a small effect size. Empathy varied more noticeably, with Copilot scoring highest and Doubao and Perplexity scoring lower. Perplexity achieved the highest DISCERN and JAMA scores, while Copilot performed best on EQIP. GQS differences were small. Readability differed significantly across models, with Gemini generally producing the most readable responses; however, all models exceeded the recommended sixth-grade reading level, and all Flesch Reading Ease Scores were below the recommended threshold. Conclusion: No single LLM performed best across all domains. Although the evaluated models generally provided safe and accurate responses to DR patient questions, limitations remained in safety escalation, empathy, transparency, and readability. LLM-generated DR information may support general patient education but should not replace professional ophthalmic consultation.
SN - 2296-2565
ER -