Completeness and guideline concordance of artificial intelligence chatbot responses to patient questions on regenerative therapies for erectile dysfunction: a multi-platform analysis.
- 1Department of Urology, Baylor College of Medicine, Houston, TX, USA.
Abstract
Interest in regenerative therapies for erectile dysfunction (ED), including stem cell therapy (SCT), low-intensity shockwave therapy (LiSWT), and platelet-rich plasma (PRP), has increased despite limited evidence and investigational designations from professional societies. Concurrently, patients increasingly use artificial intelligence (AI) large language model (LLM) chatbots to obtain health information. We evaluated the completeness and guideline concordance of AI-generated responses to patient-style questions about regenerative ED therapies, and their consistency across repeated queries. We performed a cross-sectional evaluation of four general-purpose LLM platforms: ChatGPT-4o, Gemini 2.5, Claude 3.7, and Copilot. Six patient-style prompts addressing general inquiry, evidence appraisal, safety, and treatment-seeking behavior were submitted for each therapy (SCT, LiSWT, PRP) in two independent sessions on separate days, yielding 144 responses. The primary outcome was the presence of nine prespecified counseling elements using a binary checklist. The secondary outcome was concordance with the Sexual Medicine Society of North America (SMSNA) position statement, assessed across efficacy framing, recommendation against routine use, and regulatory-status accuracy. Within-platform consistency across two sessions was assessed using Cohen's kappa. Most responses identified investigational or non-FDA-approved status (80.6%), recommended physician consultation (76.4%), and acknowledged limitations in current research (75%). References to society guidelines (14.6%), safety profiles (38.9%), alternative therapies (23.6%), and cost (31.9%) were uncommon. Peer-reviewed citations were present in fewer than half of responses (47.9%), despite frequent references to "research". Among the 130 responses that characterized efficacy, only 36% conveyed the guideline-concordant position that clinical efficacy has not been established for regenerative therapies. The majority of responses instead framed efficacy optimistically despite the lack of robust efficacy evidence. An explicit recommendation against routine clinical use appeared in only 18.8% of responses. Within-platform consistency was substantial for Copilot, Claude, and Gemini (κ=0.61-0.67) and moderate for ChatGPT (κ=0.54). General-purpose AI chatbots provide incomplete counseling regarding regenerative ED therapies and inconsistently reflect the core elements of the SMSNA position statement. Chatbot responses specifically contain gaps in disclosing safety profiles, approved alternatives, and cost. Deliberate integration of evidence-based guidance into consumer-facing LLMs is needed to prevent AI-driven misconceptions. Clinicians should assess AI use and address AI-derived expectations during clinical encounters.
Related
- Theory of Artificial Intelligence Adaptation for Linguistic Concordance: Bridging Gaps in Patient-Centered Communication
- Development and validation of an artificial intelligence model for basal cell carcinoma detection and section completeness assessment in Mohs micrographic surgery in an Asian cohort
- Diagnostic Concordance Across Text and Image Workflows in Nine Consumer Multimodal Artificial Intelligence Systems for Cutaneous Vascular Anomalies: A Cross-Sectional Audit.
- External validation of artificial intelligence-guided electrocardiogram (AI-ECG) in detecting asymptomatic left ventricular dysfunction: A systematic review and meta-analysis
- A focused PubMed artificial intelligence search program creates better literature reviews than general purpose artificial intelligence and is comparable to trainees in shoulder and elbow surgery
BibTeX
@article{low2026completeness,
title = {{Completeness and guideline concordance of artificial intelligence chatbot responses to patient questions on regenerative therapies for erectile dysfunction: a multi-platform analysis.}},
author = {Ethan L Low and Jamie Welsh and Evan Burchfiel and Shane Kronstedt and Daisy Gracia and Beatriz Hernandez and Amelia Oppenheimer and Mohit Khera},
journal = {Translational andrology and urology},
year = {2026},
month = sep,
doi = {10.21037/tau-2026-0399},
url = {https://europepmc.org/article/MED/42997193},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Completeness and guideline concordance of artificial intelligence chatbot responses to patient questions on regenerative therapies for erectile dysfunction: a multi-platform analysis.
AU - Ethan L Low
AU - Jamie Welsh
AU - Evan Burchfiel
AU - Shane Kronstedt
AU - Daisy Gracia
AU - Beatriz Hernandez
AU - Amelia Oppenheimer
AU - Mohit Khera
PY - 2026
DA - 2026/09/28
JO - Translational andrology and urology
DO - 10.21037/tau-2026-0399
UR - https://europepmc.org/article/MED/42997193
AB - Interest in regenerative therapies for erectile dysfunction (ED), including stem cell therapy (SCT), low-intensity shockwave therapy (LiSWT), and platelet-rich plasma (PRP), has increased despite limited evidence and investigational designations from professional societies. Concurrently, patients increasingly use artificial intelligence (AI) large language model (LLM) chatbots to obtain health information. We evaluated the completeness and guideline concordance of AI-generated responses to patient-style questions about regenerative ED therapies, and their consistency across repeated queries. We performed a cross-sectional evaluation of four general-purpose LLM platforms: ChatGPT-4o, Gemini 2.5, Claude 3.7, and Copilot. Six patient-style prompts addressing general inquiry, evidence appraisal, safety, and treatment-seeking behavior were submitted for each therapy (SCT, LiSWT, PRP) in two independent sessions on separate days, yielding 144 responses. The primary outcome was the presence of nine prespecified counseling elements using a binary checklist. The secondary outcome was concordance with the Sexual Medicine Society of North America (SMSNA) position statement, assessed across efficacy framing, recommendation against routine use, and regulatory-status accuracy. Within-platform consistency across two sessions was assessed using Cohen's kappa. Most responses identified investigational or non-FDA-approved status (80.6%), recommended physician consultation (76.4%), and acknowledged limitations in current research (75%). References to society guidelines (14.6%), safety profiles (38.9%), alternative therapies (23.6%), and cost (31.9%) were uncommon. Peer-reviewed citations were present in fewer than half of responses (47.9%), despite frequent references to "research". Among the 130 responses that characterized efficacy, only 36% conveyed the guideline-concordant position that clinical efficacy has not been established for regenerative therapies. The majority of responses instead framed efficacy optimistically despite the lack of robust efficacy evidence. An explicit recommendation against routine clinical use appeared in only 18.8% of responses. Within-platform consistency was substantial for Copilot, Claude, and Gemini (κ=0.61-0.67) and moderate for ChatGPT (κ=0.54). General-purpose AI chatbots provide incomplete counseling regarding regenerative ED therapies and inconsistently reflect the core elements of the SMSNA position statement. Chatbot responses specifically contain gaps in disclosing safety profiles, approved alternatives, and cost. Deliberate integration of evidence-based guidance into consumer-facing LLMs is needed to prevent AI-driven misconceptions. Clinicians should assess AI use and address AI-derived expectations during clinical encounters.
SN - 2223-4691
ER -