Artificial intelligence chatbots as information sources for herpes zoster: an evaluation of patient-education suitability, overall quality, and readability
- 1Department of Pharmacy, The Fourth Hospital of Changsha (Integrated Traditional Chinese and Western Medicine Hospital of Changsha)
- 2International Advanced Technology Application Promotion Center (Hefei)
- 3Department of Dermatology
Abstract
Objective: Clarify the effects of large-language-model types and health-education content categories on generated-text quality (patient-education suitability, overall quality) and readability, so as to provide empirical evidence for standardized AI-assisted health-communication applications.
Methods: This cross-sectional study was conducted between July 10 and July 15, 2026. Twenty herpes zoster-related questions were developed by a dermatologist and a pharmacist and categorized into five themes. The questions were then entered into five mainstream large language models (GPT-5, Doubao, DeepSeek, Qianwen, and Gemini), generating 100 textual responses. Response quality was assessed using the Chinese version of the Patient Education Materials Assessment Tool for Printable Materials (C-PEMAT-P) and Global Quality Score (GQS). Readability was evaluated via seven indices including the Automated Readability Index (ARI) and Flesch Reading Ease Score (FRES). Readability outcomes were compared using one-way repeated-measures analysis of variance with Greenhouse–Geisser correction. C-PEMAT-P raw totals and ordinal GQS scores were compared using Friedman tests. Post-hoc comparisons were adjusted using the Holm method. Category-level comparisons were performed using linear mixed-effects models with likelihood-ratio χ ² tests for continuous outcomes and an ordinal generalized estimating equation with a Wald χ ² test for GQS.
Results: Significant overall differences among models were observed for all seven readability indices and both assessment scores (all p < 0.001). Median C-PEMAT-P raw total scores were 12.50, 11.00, 10.00, 9.00 and 9.00 for GPT-5, Doubao, DeepSeek, Qianwen and Gemini, respectively; corresponding median GQS scores were 5.00, 4.00, 4.00, 3.00, and 3.00. After Holm correction, GPT-5 achieved higher scores than each of the other models on both assessment measures. However, no significant difference in C-PEMAT-P raw totals was found between DeepSeek and either Qianwen or Gemini, or between Doubao and DeepSeek, after adjustment (all adjusted p = 0.062), and Qianwen and Gemini did not differ significantly (adjusted p = 0.766). The overall effect sizes were Kendall’s W = 0.611 for C-PEMAT-P and W = 0.779 for GQS. Exploratory pooled Spearman correlation analysis revealed a positive association between the two assessment scores (ρ = 0.853).
Conclusion: Model type is a key determinant of text quality. Readability was independently influenced by both model type and content category, and the two are relatively independent. In clinical applications, GPT-5, DeepSeek, and Doubao are preferred. Readability can be optimized by integrating topic characteristics and patients’ health literacy to enhance communication accuracy. Limitations exist in this study regarding sample coverage and evaluation dimensions (the absence of subjective experience data); larger-scale verification should be conducted in future research.
Related
- Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives
- Quality, transparency, and readability of artificial intelligence chatbot responses to fibrous dysplasia-related searches: a cross-sectional comparative study
- Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability
- Are patient education materials for scalp cooling therapy accessible? A readability and suitability perspective.
- AI-generated high-intensity interval training information: a cross-sectional comparative evaluation of quality and readability
BibTeX
@article{zhang2026artificial,
title = {{Artificial intelligence chatbots as information sources for herpes zoster: an evaluation of patient-education suitability, overall quality, and readability}},
author = {Fengyu Zhang and Pengfei Tong and Hucai Yang},
journal = {Frontiers in Public Health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1961492},
url = {https://www.frontiersin.org/articles/10.3389/fpubh.2026.1961492/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Artificial intelligence chatbots as information sources for herpes zoster: an evaluation of patient-education suitability, overall quality, and readability
AU - Fengyu Zhang
AU - Pengfei Tong
AU - Hucai Yang
PY - 2026
DA - 2026/10/27
JO - Frontiers in Public Health
DO - 10.3389/fpubh.2026.1961492
UR - https://www.frontiersin.org/articles/10.3389/fpubh.2026.1961492/full
AB - Objective: Clarify the effects of large-language-model types and health-education content categories on generated-text quality (patient-education suitability, overall quality) and readability, so as to provide empirical evidence for standardized AI-assisted health-communication applications. Methods: This cross-sectional study was conducted between July 10 and July 15, 2026. Twenty herpes zoster-related questions were developed by a dermatologist and a pharmacist and categorized into five themes. The questions were then entered into five mainstream large language models (GPT-5, Doubao, DeepSeek, Qianwen, and Gemini), generating 100 textual responses. Response quality was assessed using the Chinese version of the Patient Education Materials Assessment Tool for Printable Materials (C-PEMAT-P) and Global Quality Score (GQS). Readability was evaluated via seven indices including the Automated Readability Index (ARI) and Flesch Reading Ease Score (FRES). Readability outcomes were compared using one-way repeated-measures analysis of variance with Greenhouse–Geisser correction. C-PEMAT-P raw totals and ordinal GQS scores were compared using Friedman tests. Post-hoc comparisons were adjusted using the Holm method. Category-level comparisons were performed using linear mixed-effects models with likelihood-ratio χ ² tests for continuous outcomes and an ordinal generalized estimating equation with a Wald χ ² test for GQS. Results: Significant overall differences among models were observed for all seven readability indices and both assessment scores (all p < 0.001). Median C-PEMAT-P raw total scores were 12.50, 11.00, 10.00, 9.00 and 9.00 for GPT-5, Doubao, DeepSeek, Qianwen and Gemini, respectively; corresponding median GQS scores were 5.00, 4.00, 4.00, 3.00, and 3.00. After Holm correction, GPT-5 achieved higher scores than each of the other models on both assessment measures. However, no significant difference in C-PEMAT-P raw totals was found between DeepSeek and either Qianwen or Gemini, or between Doubao and DeepSeek, after adjustment (all adjusted p = 0.062), and Qianwen and Gemini did not differ significantly (adjusted p = 0.766). The overall effect sizes were Kendall’s W = 0.611 for C-PEMAT-P and W = 0.779 for GQS. Exploratory pooled Spearman correlation analysis revealed a positive association between the two assessment scores (ρ = 0.853). Conclusion: Model type is a key determinant of text quality. Readability was independently influenced by both model type and content category, and the two are relatively independent. In clinical applications, GPT-5, DeepSeek, and Doubao are preferred. Readability can be optimized by integrating topic characteristics and patients’ health literacy to enhance communication accuracy. Limitations exist in this study regarding sample coverage and evaluation dimensions (the absence of subjective experience data); larger-scale verification should be conducted in future research.
SN - 2296-2565
ER -