Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
- 1Department of Colorectal Surgery, Jintang Hospital Jintang First People's Hospital, West China Hospital Sichuan University , , ,
- 2Graduate School of Tianjin University of Traditional Chinese Medicine , ,
- 3Department of Integrated Traditional Chinese and Western Medicine Proctology, Affiliated Hospital of North Sichuan Medical College , , ,
Abstract
Background
Rectal prolapse (RP) is a pelvic floor disorder that can cause rectal protrusion, defecatory dysfunction, and fecal incontinence. Patients increasingly use large language models (LLMs) for health information, yet the reliability and readability of LLM responses to patient-oriented questions about RP remain unclear.Objective
To compare the reliability and readability of five LLM platform-model configurations in answering English-language patient-oriented questions about RP.Methods
In this web-based cross-sectional study, worldwide Google Trends queries from the preceding 5 years were identified using the MeSH term "rectal prolapse." Two colorectal specialists screened the queries and standardized them into 15 patient-oriented questions. The questions were submitted to ChatGPT, Gemini, Microsoft Copilot, Perplexity, and DeepSeek under standardized single-turn, zero-shot conditions, with one response collected per question from each platform-model configuration. Two independent evaluators assessed the responses using DISCERN, the Ensuring Quality Information for Patients (EQIP) instrument, the Global Quality Score (GQS), and the JAMA benchmark criteria. Using evidence-based guidelines and question-specific checklists of key clinical content, the evaluators also independently assessed clinical factual accuracy and completeness and identified potentially harmful recommendations and critical omissions. Readability was evaluated using the Automated Readability Index (ARI), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CL), Simple Measure of Gobbledygook (SMOG), and Flesch Reading Ease Score (FRES).Results
DISCERN (P = 0.001), EQIP (P < 0.001), and JAMA scores (P < 0.001) differed across configurations, whereas GQS did not (P = 0.175). ChatGPT and Copilot had the highest median DISCERN scores (44.00 for both), and DeepSeek had the lowest (35.00); ChatGPT and Copilot each scored significantly higher than DeepSeek. Copilot had the highest median EQIP score (80.00), followed by Perplexity (75.00). Median JAMA scores were 1.00 for ChatGPT, Copilot, and Perplexity and 0.00 for Gemini and DeepSeek. All five models had a median GQS of 4.00. The median clinical factual accuracy score was 5.00 (IQR, 5.00-5.00), and 69 of 75 responses (92.0%) were rated fully accurate. Median completeness was 91.7% (IQR, 87.5%-100.0%). No response contained a potentially harmful recommendation; critical omissions were found in 6 of the 50 applicable responses (12.0%). All configurations had median grade-level scores >6 and median FRES <80. No model met the prespecified sixth-grade readability standard.Conclusion
The five LLMs generally produced clinically accurate information about RP. No potentially harmful recommendations were identified. Yet information quality and source transparency differed among models, some responses omitted critical clinical information, and readability was poor across all five. LLMs are currently better suited as professionally reviewed adjuncts to patient education than as substitutes for specialist assessment or individualized clinical decision-making.
Related
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Performance of large language models in postoperative hip fracture rehabilitation counseling for older adults: a comparative evaluation of safety, accuracy, reliability, readability, and empathy
BibTeX
@article{zhang2026reliability,
title = {{Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study}},
author = {Le Zhang and Lei Liang and Xin Peng and Shihong Liu and Lingli Zhong and Zhenyu Li},
journal = {Frontiers in medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1971190},
url = {https://europepmc.org/article/PMC/PMC13717504},
note = {CC BY},
}
RIS
TY - JOUR
TI - Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
AU - Le Zhang
AU - Lei Liang
AU - Xin Peng
AU - Shihong Liu
AU - Lingli Zhong
AU - Zhenyu Li
PY - 2026
DA - 2026/10/14
JO - Frontiers in medicine
DO - 10.3389/fmed.2026.1971190
UR - https://europepmc.org/article/PMC/PMC13717504
AB - Background Rectal prolapse (RP) is a pelvic floor disorder that can cause rectal protrusion, defecatory dysfunction, and fecal incontinence. Patients increasingly use large language models (LLMs) for health information, yet the reliability and readability of LLM responses to patient-oriented questions about RP remain unclear.Objective To compare the reliability and readability of five LLM platform-model configurations in answering English-language patient-oriented questions about RP.Methods In this web-based cross-sectional study, worldwide Google Trends queries from the preceding 5 years were identified using the MeSH term "rectal prolapse." Two colorectal specialists screened the queries and standardized them into 15 patient-oriented questions. The questions were submitted to ChatGPT, Gemini, Microsoft Copilot, Perplexity, and DeepSeek under standardized single-turn, zero-shot conditions, with one response collected per question from each platform-model configuration. Two independent evaluators assessed the responses using DISCERN, the Ensuring Quality Information for Patients (EQIP) instrument, the Global Quality Score (GQS), and the JAMA benchmark criteria. Using evidence-based guidelines and question-specific checklists of key clinical content, the evaluators also independently assessed clinical factual accuracy and completeness and identified potentially harmful recommendations and critical omissions. Readability was evaluated using the Automated Readability Index (ARI), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CL), Simple Measure of Gobbledygook (SMOG), and Flesch Reading Ease Score (FRES).Results DISCERN (P = 0.001), EQIP (P < 0.001), and JAMA scores (P < 0.001) differed across configurations, whereas GQS did not (P = 0.175). ChatGPT and Copilot had the highest median DISCERN scores (44.00 for both), and DeepSeek had the lowest (35.00); ChatGPT and Copilot each scored significantly higher than DeepSeek. Copilot had the highest median EQIP score (80.00), followed by Perplexity (75.00). Median JAMA scores were 1.00 for ChatGPT, Copilot, and Perplexity and 0.00 for Gemini and DeepSeek. All five models had a median GQS of 4.00. The median clinical factual accuracy score was 5.00 (IQR, 5.00-5.00), and 69 of 75 responses (92.0%) were rated fully accurate. Median completeness was 91.7% (IQR, 87.5%-100.0%). No response contained a potentially harmful recommendation; critical omissions were found in 6 of the 50 applicable responses (12.0%). All configurations had median grade-level scores >6 and median FRES <80. No model met the prespecified sixth-grade readability standard.Conclusion The five LLMs generally produced clinically accurate information about RP. No potentially harmful recommendations were identified. Yet information quality and source transparency differed among models, some responses omitted critical clinical information, and readability was poor across all five. LLMs are currently better suited as professionally reviewed adjuncts to patient education than as substitutes for specialist assessment or individualized clinical decision-making.
SN - 2296-858X
ER -