Generative AI chatbots as sources of phimosis-related consumer health information: a cross-sectional comparison of quality, source transparency, and readability.
- 1Department of Urology, Nanchang People's Hospital, Nanchang, China.
Abstract
Background
Publicly accessible generative artificial intelligence chatbot products are increasingly used for sensitive health questions, but the quality, source transparency, and readability of phimosis-related information remain uncertain.Methods
We conducted an internet-based cross-sectional comparative study on 1 July 2026. 15 of 25 Google Trends candidate terms were submitted unchanged to ChatGPT (displayed as GPT-5.5), Microsoft Copilot, Gemini (displayed as 3.5 Flash), and Perplexity, yielding 60 responses. Two practising urologists holding senior professional titles, each with more than 10 years of clinical experience, independently assessed responses using DISCERN, Ensuring Quality Information for Patients (EQIP), the Global Quality Score (GQS), and Journal of the American Medical Association (JAMA) benchmark criteria. Inter-rater reliability was assessed using ICC (2,1) for DISCERN and EQIP and quadratic-weighted Cohen's kappa for JAMA and GQS. Because the same 15 queries were evaluated across all four products, quality outcomes were analysed as matched data using Friedman tests with Kendall's W and Holm-adjusted paired Wilcoxon signed-rank comparisons. Readability was assessed with six established indices.Results
Inter-rater reliability was good to almost perfect (coefficients 0.866-0.882). After Holm adjustment across the four omnibus tests, differences remained for DISCERN (p = 0.007; W = 0.301), EQIP (p < 0.001; W = 0.587), JAMA (p < 0.001; W = 0.953), and GQS (p = 0.049; W = 0.174). For DISCERN, Copilot scored higher than ChatGPT and Gemini, and Perplexity scored higher than Gemini. For EQIP and JAMA, Copilot and Perplexity scored higher than ChatGPT and Gemini. No GQS pairwise comparison remained significant after Holm correction. None of the 15 responses from any product met any individual prespecified readability threshold. A sensitivity analysis excluding the broad standalone prompts 'circumcision' and 'balanitis' preserved the DISCERN, EQIP, and JAMA findings but not the GQS omnibus difference.Conclusion
In this purposively derived 15-query, single-date snapshot, chatbot products differed in selected measures of information quality and source transparency, while readability remained consistently above the prespecified sixth-grade target. These findings are conditional on the sampled queries and outputs and should not be interpreted as stable platform-wide estimates of clinical accuracy or safety.
Related
- Evaluating AI chatbots as sources of public-facing encephalitis health information: a cross-sectional comparative study of quality, transparency, and readability
- Quality, transparency, and readability of artificial intelligence chatbot responses to fibrous dysplasia-related searches: a cross-sectional comparative study
- Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study
- Accuracy, Quality and Readability of Knee Osteoarthritis Information on Social Media (Instagram e TikTok) and Search Platforms (Google e YouTube): A Cross-Sectional Study.
- Quality indicators, reliability, transparency and content completeness of short-video information on traditional Chinese medicine for insomnia across three major Chinese platforms: a cross-sectional study
BibTeX
@article{luo2026generative,
title = {{Generative AI chatbots as sources of phimosis-related consumer health information: a cross-sectional comparison of quality, source transparency, and readability.}},
author = {Qinghua Luo and Leihua Cao and Pan Ding},
journal = {Frontiers in public health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1983152},
url = {https://europepmc.org/article/MED/42898506},
note = {CC BY},
}
RIS
TY - JOUR
TI - Generative AI chatbots as sources of phimosis-related consumer health information: a cross-sectional comparison of quality, source transparency, and readability.
AU - Qinghua Luo
AU - Leihua Cao
AU - Pan Ding
PY - 2026
DA - 2026/10/05
JO - Frontiers in public health
DO - 10.3389/fpubh.2026.1983152
UR - https://europepmc.org/article/MED/42898506
AB - Background Publicly accessible generative artificial intelligence chatbot products are increasingly used for sensitive health questions, but the quality, source transparency, and readability of phimosis-related information remain uncertain.Methods We conducted an internet-based cross-sectional comparative study on 1 July 2026. 15 of 25 Google Trends candidate terms were submitted unchanged to ChatGPT (displayed as GPT-5.5), Microsoft Copilot, Gemini (displayed as 3.5 Flash), and Perplexity, yielding 60 responses. Two practising urologists holding senior professional titles, each with more than 10 years of clinical experience, independently assessed responses using DISCERN, Ensuring Quality Information for Patients (EQIP), the Global Quality Score (GQS), and Journal of the American Medical Association (JAMA) benchmark criteria. Inter-rater reliability was assessed using ICC (2,1) for DISCERN and EQIP and quadratic-weighted Cohen's kappa for JAMA and GQS. Because the same 15 queries were evaluated across all four products, quality outcomes were analysed as matched data using Friedman tests with Kendall's W and Holm-adjusted paired Wilcoxon signed-rank comparisons. Readability was assessed with six established indices.Results Inter-rater reliability was good to almost perfect (coefficients 0.866-0.882). After Holm adjustment across the four omnibus tests, differences remained for DISCERN (p = 0.007; W = 0.301), EQIP (p < 0.001; W = 0.587), JAMA (p < 0.001; W = 0.953), and GQS (p = 0.049; W = 0.174). For DISCERN, Copilot scored higher than ChatGPT and Gemini, and Perplexity scored higher than Gemini. For EQIP and JAMA, Copilot and Perplexity scored higher than ChatGPT and Gemini. No GQS pairwise comparison remained significant after Holm correction. None of the 15 responses from any product met any individual prespecified readability threshold. A sensitivity analysis excluding the broad standalone prompts 'circumcision' and 'balanitis' preserved the DISCERN, EQIP, and JAMA findings but not the GQS omnibus difference.Conclusion In this purposively derived 15-query, single-date snapshot, chatbot products differed in selected measures of information quality and source transparency, while readability remained consistently above the prespecified sixth-grade target. These findings are conditional on the sampled queries and outputs and should not be interpreted as stable platform-wide estimates of clinical accuracy or safety.
SN - 2296-2565
ER -