Performance of five large language models in frozen shoulder patient education: a multidimensional evaluation of readability, quality, clinical alignment, and safety
Abstract
This study compared the educational suitability, overall quality, translation-based readability, clinical intent alignment, and safety of frozen shoulder patient education materials generated by GPT-5, DeepSeek, Doubao, Wenxin Yiyan, and Tongyi Qianwen. Twenty clinician-curated questions covering five content categories were submitted once to each model, producing 100 first-pass responses. C-PEMAT and GQS assessed educational quality, seven English readability formulas were applied to consensus translations of the Chinese outputs, and two preliminary clinician-developed measures assessed clinical key-point coverage and safety-related caution. C-PEMAT and GQS differed significantly among models (both P < 0.001), with GPT-5 obtaining the highest scores, followed by DeepSeek and Doubao. Quality scores did not differ significantly across content categories, whereas all readability indices did. C-PEMAT and GQS were moderately correlated ( r = 0.68), but quality measures were weakly associated with most readability indices. GPT-5 and DeepSeek showed relatively favorable Clinical Intent Alignment scores, while GPT-5 had the highest descriptive Clinical Safety Score. These findings are specific to the tested access date, platforms, prompts, and model versions. LLM-generated materials may serve as auxiliary drafts, but clinician review remains necessary for medical accuracy, safety, readability, and patient-specific appropriateness.
Related
- Multidimensional evaluation of generative AI chatbots for public consultation on sudden cardiac death: safety, accuracy, empathy, reliability, quality, and readability across six models
- Enhancing patient understanding of ovarian cancer: creation, validation and user-readability evaluation of an education booklet
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Patient-facing large language model assistants for chronic constipation education: A comparative study of expert evaluation, readability, and patient acceptability
- Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
BibTeX
@article{liang2026performance,
title = {{Performance of five large language models in frozen shoulder patient education: a multidimensional evaluation of readability, quality, clinical alignment, and safety}},
author = {Jiachen Liang and Wanying Gui and Huanan Li},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-66627-6},
url = {https://www.nature.com/articles/s41598-026-66627-6},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Performance of five large language models in frozen shoulder patient education: a multidimensional evaluation of readability, quality, clinical alignment, and safety
AU - Jiachen Liang
AU - Wanying Gui
AU - Huanan Li
PY - 2026
DA - 2026/10/05
JO - Scientific Reports
DO - 10.1038/s41598-026-66627-6
UR - https://www.nature.com/articles/s41598-026-66627-6
AB - This study compared the educational suitability, overall quality, translation-based readability, clinical intent alignment, and safety of frozen shoulder patient education materials generated by GPT-5, DeepSeek, Doubao, Wenxin Yiyan, and Tongyi Qianwen. Twenty clinician-curated questions covering five content categories were submitted once to each model, producing 100 first-pass responses. C-PEMAT and GQS assessed educational quality, seven English readability formulas were applied to consensus translations of the Chinese outputs, and two preliminary clinician-developed measures assessed clinical key-point coverage and safety-related caution. C-PEMAT and GQS differed significantly among models (both P < 0.001), with GPT-5 obtaining the highest scores, followed by DeepSeek and Doubao. Quality scores did not differ significantly across content categories, whereas all readability indices did. C-PEMAT and GQS were moderately correlated ( r = 0.68), but quality measures were weakly associated with most readability indices. GPT-5 and DeepSeek showed relatively favorable Clinical Intent Alignment scores, while GPT-5 had the highest descriptive Clinical Safety Score. These findings are specific to the tested access date, platforms, prompts, and model versions. LLM-generated materials may serve as auxiliary drafts, but clinician review remains necessary for medical accuracy, safety, readability, and patient-specific appropriateness.
SN - 2045-2322
ER -