Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
- 1Department of General Surgery, İzmir Atatürk Training and Research Hospital, İzmir Katip Çelebi University, İzmir, Turkey
Abstract
Background: Breast cancer, the most common malignancy among women, remains a major global public health concern. With the rapid growth of artificial intelligence–based language models, it is essential to evaluate their potential roles in patient education. This study compared ChatGPT and Gemini in responding to patient-oriented questions about breast cancer and its surgical treatment regarding scientific accuracy, clarity, and unnecessary detail. Methods: Forty frequently asked questions were collected from Turkish online sources. Both models were queried under identical conditions on August 13, 2025, and the responses were anonymized for blinded evaluation. Four general surgeons experienced in breast surgery independently assessed each response using a five-point Likert scale across three domains: scientific accuracy, clarity, and unnecessary detail. Results: Response lengths were recorded and compared. ChatGPT achieved significantly higher median scores than Gemini in scientific accuracy [4.75 (3.50–5.00) vs. 4.25 (3.50–5.00); p < 0.001], clarity [4.75 (3.50–5.00) vs. 4.25 (3.25–5.00); p = 0.005], and unnecessary detail [5.00 (5.00–5.00) vs. 4.50 (3.50–5.00); p < 0.001]. The overall median score was 4.83 (4.08–5.00) for ChatGPT and 4.33 (3.50–4.92) for Gemini (p < 0.001). Gemini’s responses were significantly longer (244 ± 84 vs. 170 ± 47 words; p < 0.001). Conclusion: Both models demonstrated acceptable overall performance. However, ChatGPT’s higher scores and shorter, more focused responses indicate that it may be a more efficient tool for addressing breast cancer–related patient questions. In their current form, these models should not be used for diagnostic or clinical decision-making purposes. AI-generated health information should be used only under expert supervision and for educational or supportive purposes.
Related
- Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
- Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Comparing ChatGPT and physicians’ answers to endometriosis questions on Reddit: A blind expert evaluation
- Assessing the reliability of AI-generated responses to patient-oriented lung cancer questions: a multi-platform study.
BibTeX
@article{bozok2026comparative,
title = {{Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer}},
author = {Yeliz Yilmaz Bozok and Nihan Acar and Murat Kemal Atahan and Cem Karaali},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0358325},
url = {https://doi.org/10.1371/journal.pone.0358325},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
AU - Yeliz Yilmaz Bozok
AU - Nihan Acar
AU - Murat Kemal Atahan
AU - Cem Karaali
PY - 2026
DA - 2026/09/16
JO - PLOS One
DO - 10.1371/journal.pone.0358325
UR - https://doi.org/10.1371/journal.pone.0358325
AB - Background: Breast cancer, the most common malignancy among women, remains a major global public health concern. With the rapid growth of artificial intelligence–based language models, it is essential to evaluate their potential roles in patient education. This study compared ChatGPT and Gemini in responding to patient-oriented questions about breast cancer and its surgical treatment regarding scientific accuracy, clarity, and unnecessary detail. Methods: Forty frequently asked questions were collected from Turkish online sources. Both models were queried under identical conditions on August 13, 2025, and the responses were anonymized for blinded evaluation. Four general surgeons experienced in breast surgery independently assessed each response using a five-point Likert scale across three domains: scientific accuracy, clarity, and unnecessary detail. Results: Response lengths were recorded and compared. ChatGPT achieved significantly higher median scores than Gemini in scientific accuracy [4.75 (3.50–5.00) vs. 4.25 (3.50–5.00); p < 0.001], clarity [4.75 (3.50–5.00) vs. 4.25 (3.25–5.00); p = 0.005], and unnecessary detail [5.00 (5.00–5.00) vs. 4.50 (3.50–5.00); p < 0.001]. The overall median score was 4.83 (4.08–5.00) for ChatGPT and 4.33 (3.50–4.92) for Gemini (p < 0.001). Gemini’s responses were significantly longer (244 ± 84 vs. 170 ± 47 words; p < 0.001). Conclusion: Both models demonstrated acceptable overall performance. However, ChatGPT’s higher scores and shorter, more focused responses indicate that it may be a more efficient tool for addressing breast cancer–related patient questions. In their current form, these models should not be used for diagnostic or clinical decision-making purposes. AI-generated health information should be used only under expert supervision and for educational or supportive purposes.
SN - 1932-6203
ER -