Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study
Abstract
Objective To compare three large language model (LLM) systems answering patient questions about renal cystic lesions in terms of medical content quality and radiologist-rated patient understandability. We also assessed inter-rater reliability and, exploratorily, the readability characteristics of English translations. Methods This cross-sectional, blinded, paired, multi-reader evaluation included 40 questions derived from authoritative patient education resources and web searches. Each question was submitted to OpenAI GPT-5.5, DeepSeek V4 Pro, and Ant A-Fu. Three radiologists independently rated every response on six 1–5 dimensions: medical accuracy, completeness, safety/risk guidance, radiologist-rated patient understandability, actionability/care advice, and overall quality. Overall comparisons used the Friedman test. Pairwise comparisons used paired Wilcoxon signed-rank tests with Holm correction. Intraclass correlation coefficients (ICCs) quantified inter-rater reliability. Sensitivity analyses evaluated all three responses and included 10,000 simulated random selections. Results No statistically significant overall differences were found in medical accuracy (4.65/4.68/4.65, P=0.790) or completeness (4.58/4.67/4.68, P=0.362). Safety/risk guidance also did not differ significantly (4.65/4.50/4.57, P=0.651). Differences were nonsignificant for actionability/care advice (4.34/4.42/4.38, P=0.120) and overall quality (4.30/4.38/4.40, P=0.134). Radiologist-rated patient understandability was the only dimension with an overall between-system difference (4.05/4.12/4.41, P<0.001; Kendall’s W=0.23). Ant A-Fu scored higher than OpenAI GPT-5.5 (Holm-adjusted P=0.0039; rank-biserial r=−0.768) and DeepSeek V4 Pro (Holm-adjusted P=0.0138; r=−0.638). This dimension showed very high inter-rater reliability [ICC(2,1)=0.947; ICC(2,3)=0.982]. The patient-understandability difference remained significant when all three responses were included and in all 10,000 random-selection simulations; the other five dimensions remained nonsignificant. Conclusion The primary analysis did not detect clear between-system differences in most medical-content dimensions. However, pronounced high-score clustering limited the discriminatory ability of the five-point rubric, and these nonsignificant findings do not establish similar or equivalent medical-content performance. Radiologist-rated patient understandability differed between systems and showed very high inter-rater reliability. Prospective studies involving patients are needed to determine whether this professional proxy predicts actual patient comprehension.
Related
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Performance of large language models in answering orthodontic questions: A scoping review.
- Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
- Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non‐English Languages
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
BibTeX
@article{liu2026medical,
title = {{Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study}},
author = {Zhenghan Liu MM and Qianqian Chen MM and Tianxiao Hu BM and Mingyu Li MM and Haomei Yu MM and Jie Wang MD, PhD and Xiangming Fang MD, PhD and Xiaoyun Hu MD, PhD},
journal = {Digital Health},
year = {2026},
month = oct,
doi = {10.1177/20552076261504852},
url = {https://doi.org/10.1177/20552076261504852},
note = {CC BY},
}
RIS
TY - JOUR
TI - Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study
AU - Zhenghan Liu MM
AU - Qianqian Chen MM
AU - Tianxiao Hu BM
AU - Mingyu Li MM
AU - Haomei Yu MM
AU - Jie Wang MD, PhD
AU - Xiangming Fang MD, PhD
AU - Xiaoyun Hu MD, PhD
PY - 2026
DA - 2026/10/01
JO - Digital Health
DO - 10.1177/20552076261504852
UR - https://doi.org/10.1177/20552076261504852
AB - Objective To compare three large language model (LLM) systems answering patient questions about renal cystic lesions in terms of medical content quality and radiologist-rated patient understandability. We also assessed inter-rater reliability and, exploratorily, the readability characteristics of English translations. Methods This cross-sectional, blinded, paired, multi-reader evaluation included 40 questions derived from authoritative patient education resources and web searches. Each question was submitted to OpenAI GPT-5.5, DeepSeek V4 Pro, and Ant A-Fu. Three radiologists independently rated every response on six 1–5 dimensions: medical accuracy, completeness, safety/risk guidance, radiologist-rated patient understandability, actionability/care advice, and overall quality. Overall comparisons used the Friedman test. Pairwise comparisons used paired Wilcoxon signed-rank tests with Holm correction. Intraclass correlation coefficients (ICCs) quantified inter-rater reliability. Sensitivity analyses evaluated all three responses and included 10,000 simulated random selections. Results No statistically significant overall differences were found in medical accuracy (4.65/4.68/4.65, P=0.790) or completeness (4.58/4.67/4.68, P=0.362). Safety/risk guidance also did not differ significantly (4.65/4.50/4.57, P=0.651). Differences were nonsignificant for actionability/care advice (4.34/4.42/4.38, P=0.120) and overall quality (4.30/4.38/4.40, P=0.134). Radiologist-rated patient understandability was the only dimension with an overall between-system difference (4.05/4.12/4.41, P<0.001; Kendall’s W=0.23). Ant A-Fu scored higher than OpenAI GPT-5.5 (Holm-adjusted P=0.0039; rank-biserial r=−0.768) and DeepSeek V4 Pro (Holm-adjusted P=0.0138; r=−0.638). This dimension showed very high inter-rater reliability [ICC(2,1)=0.947; ICC(2,3)=0.982]. The patient-understandability difference remained significant when all three responses were included and in all 10,000 random-selection simulations; the other five dimensions remained nonsignificant. Conclusion The primary analysis did not detect clear between-system differences in most medical-content dimensions. However, pronounced high-score clustering limited the discriminatory ability of the five-point rubric, and these nonsignificant findings do not establish similar or equivalent medical-content performance. Radiologist-rated patient understandability differed between systems and showed very high inter-rater reliability. Prospective studies involving patients are needed to determine whether this professional proxy predicts actual patient comprehension.
SN - 2055-2076
ER -