Safety, accuracy, empathy and information quality of six large language models answering patient-facing questions about diabetic kidney disease: a cross-sectional comparative study
- 1School of Nursing, Hunan Medical College, Huaihua, Hunan, China
- 2Department of Nephrology, Huaihua Central Hospital, Huaihua, Hunan, China
Abstract
Background: Large language models (LLMs) may support patient education, but their suitability for diabetic kidney disease (DKD) remains uncertain. We compared six publicly accessible LLM chatbots—ChatGPT, Gemini, DeepSeek, Doubao, Kimi, and Qwen—across safety, accuracy, empathy, information quality, and readability, with safety as the primary outcome.
Methods: In this CHART-guided cross-sectional study, 42 standardized English DKD questions were submitted once to each chatbot in independent single-turn sessions, generating 252 responses. Five source-masked clinical raters assessed safety, accuracy, empathy, DISCERN, EQIP, JAMA benchmarks, and the Global Quality Scale; readability was assessed automatically. Agreement was evaluated using Fleiss'κ and ICC (2,1)/ICC (2,5). Model differences were tested using Cochran's Q or Friedman tests with Kendall's W and Benjamini–Hochberg correction. Potentially unsafe responses underwent qualitative review.
Results: Safety showed substantial agreement (κ = 0.854); ICC (2,1) ranged from 0.552 to 0.855 and ICC (2,5) from 0.860 to 0.967. Safe-response rates ranged from 83.3% to 92.9% without significant between-model differences (P = 0.726). Accuracy and empathy differed significantly (both P < 0.001), with ChatGPT highest for accuracy and Gemini highest for clinician-rated empathy. Information quality also differed significantly, with ChatGPT performing best overall and Doubao lowest. All six readability indices differed across models (all P < 0.001; Kendall's W = 0.139–0.255). DeepSeek had the lowest mean scores on all five grade-level indices and the highest mean FRES, but mean grade-level estimates exceeded the sixth-grade reference level for every model. Thirty of 252 responses (11.9%) were potentially unsafe, most often because of incomplete medication-related information or insufficiently individualized fluid advice.
Conclusions: Publicly accessible LLMs can provide plausible DKD information but show variable performance and safety risks. They may support, but should not replace, professional patient education; guideline anchoring, source verification, improved readability, and clinical oversight remain essential.
Related
- Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
- Clinical accuracy, information quality and readability of chatbot responses to patient questions about lower extremity amputation
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
BibTeX
@article{yu2026safety,
title = {{Safety, accuracy, empathy and information quality of six large language models answering patient-facing questions about diabetic kidney disease: a cross-sectional comparative study}},
author = {Shuai Yu and Ruirui Huang and Wenli Zhou and Ning Duan and Wenjie Huang},
journal = {Frontiers in Medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.2007153},
url = {https://doi.org/10.3389/fmed.2026.2007153},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Safety, accuracy, empathy and information quality of six large language models answering patient-facing questions about diabetic kidney disease: a cross-sectional comparative study
AU - Shuai Yu
AU - Ruirui Huang
AU - Wenli Zhou
AU - Ning Duan
AU - Wenjie Huang
PY - 2026
DA - 2026/10/30
JO - Frontiers in Medicine
DO - 10.3389/fmed.2026.2007153
UR - https://doi.org/10.3389/fmed.2026.2007153
AB - Background: Large language models (LLMs) may support patient education, but their suitability for diabetic kidney disease (DKD) remains uncertain. We compared six publicly accessible LLM chatbots—ChatGPT, Gemini, DeepSeek, Doubao, Kimi, and Qwen—across safety, accuracy, empathy, information quality, and readability, with safety as the primary outcome. Methods: In this CHART-guided cross-sectional study, 42 standardized English DKD questions were submitted once to each chatbot in independent single-turn sessions, generating 252 responses. Five source-masked clinical raters assessed safety, accuracy, empathy, DISCERN, EQIP, JAMA benchmarks, and the Global Quality Scale; readability was assessed automatically. Agreement was evaluated using Fleiss'κ and ICC (2,1)/ICC (2,5). Model differences were tested using Cochran's Q or Friedman tests with Kendall's W and Benjamini–Hochberg correction. Potentially unsafe responses underwent qualitative review. Results: Safety showed substantial agreement (κ = 0.854); ICC (2,1) ranged from 0.552 to 0.855 and ICC (2,5) from 0.860 to 0.967. Safe-response rates ranged from 83.3% to 92.9% without significant between-model differences (P = 0.726). Accuracy and empathy differed significantly (both P < 0.001), with ChatGPT highest for accuracy and Gemini highest for clinician-rated empathy. Information quality also differed significantly, with ChatGPT performing best overall and Doubao lowest. All six readability indices differed across models (all P < 0.001; Kendall's W = 0.139–0.255). DeepSeek had the lowest mean scores on all five grade-level indices and the highest mean FRES, but mean grade-level estimates exceeded the sixth-grade reference level for every model. Thirty of 252 responses (11.9%) were potentially unsafe, most often because of incomplete medication-related information or insufficiently individualized fluid advice. Conclusions: Publicly accessible LLMs can provide plausible DKD information but show variable performance and safety risks. They may support, but should not replace, professional patient education; guideline anchoring, source verification, improved readability, and clinical oversight remain essential.
SN - 2296-858X
ER -