Comparing ChatGPT and physicians’ answers to endometriosis questions on Reddit: A blind expert evaluation
Abstract
OBJECTIVES: To compare the perceived quality, safety, and relevance of ChatGPT responses to those provided by verified physicians on Reddit, a large online discussion platform, in response to questions related to endometriosis. METHODS: We selected 30 endometriosis-related questions posted on Reddit's r/AskDocs forum, each answered by a verified physician. Using the same question prompts, ChatGPT (GPT-3.5) generated matched-length responses. Responses were anonymized, randomized (A/B format), and assessed blindly by three university-affiliated physicians using a 11-item Likert-scale questionnaire covering medical accuracy, safety, clarity, empathy, and alignment with best practices. Evaluators also indicated which response they considered most pertinent and whether they suspected AI authorship. RESULTS: ChatGPT responses were rated significantly higher than physicians' responses on most items, including medical coherence (mean 3.89 ± 0.89 vs. 3.08 ± 0.92), clarity (3.93 ± 0.95 vs. 3.04 ± 0.99), and empathy (3.91 ± 0.93 vs. 2.76 ± 1.09), all with p-values < 0.001. Experts selected ChatGPT as the most pertinent response in 63.3 % of cases. A substantial proportion of responses from both sources were considered potentially dangerous by at least one expert: 26.7 % for ChatGPT and 60.0 % for physicians (p = 0.019). CONCLUSION: ChatGPT outperformed Reddit physicians on multiple expert-rated criteria, particularly in terms of clarity, empathy, and adherence to clinical recommendations. However, the non-negligible proportion of responses considered potentially dangerous by experts underscores the need for cautious use and appropriate supervision of such technologies.
Related
- Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
- Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
- The readability and quality paradox: Comparing ChatGPT, Gemini, and Perplexity outputs on pediatric chest pain queries
BibTeX
@article{beaulieu2026comparing,
title = {{Comparing ChatGPT and physicians’ answers to endometriosis questions on Reddit: A blind expert evaluation}},
author = {Clémence Beaulieu and Aubert Agostini and Patrice Crochet and Xavier Carcopino and Salma Touleimat and Antoine Netter},
journal = {International Journal of Medical Informatics},
year = {2026},
month = sep,
doi = {10.1016/j.ijmedinf.2025.106034},
url = {https://normandie-univ.hal.science/hal-05743636v1},
note = {CC BY},
}
RIS
TY - JOUR
TI - Comparing ChatGPT and physicians’ answers to endometriosis questions on Reddit: A blind expert evaluation
AU - Clémence Beaulieu
AU - Aubert Agostini
AU - Patrice Crochet
AU - Xavier Carcopino
AU - Salma Touleimat
AU - Antoine Netter
PY - 2026
DA - 2026/09/09
JO - International Journal of Medical Informatics
DO - 10.1016/j.ijmedinf.2025.106034
UR - https://normandie-univ.hal.science/hal-05743636v1
AB - OBJECTIVES: To compare the perceived quality, safety, and relevance of ChatGPT responses to those provided by verified physicians on Reddit, a large online discussion platform, in response to questions related to endometriosis. METHODS: We selected 30 endometriosis-related questions posted on Reddit's r/AskDocs forum, each answered by a verified physician. Using the same question prompts, ChatGPT (GPT-3.5) generated matched-length responses. Responses were anonymized, randomized (A/B format), and assessed blindly by three university-affiliated physicians using a 11-item Likert-scale questionnaire covering medical accuracy, safety, clarity, empathy, and alignment with best practices. Evaluators also indicated which response they considered most pertinent and whether they suspected AI authorship. RESULTS: ChatGPT responses were rated significantly higher than physicians' responses on most items, including medical coherence (mean 3.89 ± 0.89 vs. 3.08 ± 0.92), clarity (3.93 ± 0.95 vs. 3.04 ± 0.99), and empathy (3.91 ± 0.93 vs. 2.76 ± 1.09), all with p-values < 0.001. Experts selected ChatGPT as the most pertinent response in 63.3 % of cases. A substantial proportion of responses from both sources were considered potentially dangerous by at least one expert: 26.7 % for ChatGPT and 60.0 % for physicians (p = 0.019). CONCLUSION: ChatGPT outperformed Reddit physicians on multiple expert-rated criteria, particularly in terms of clarity, empathy, and adherence to clinical recommendations. However, the non-negligible proportion of responses considered potentially dangerous by experts underscores the need for cautious use and appropriate supervision of such technologies.
SN - 1386-5056
ER -