Skip to content
DraconiticResearch
Search Sources API
203,234 studies31 sourcesUpdated 21 min ago
Peer-reviewedCC BYJMIR dermatology

Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.

  • Elif Bal Avcı1
  • Buğra Burç Dağtaş2
  • Dilara İlhan Erdil3
  • Ozan Erdem4
  • Andaç Salman5
  • Selçuk Korkmaz6
  • Sezgi Sarıkaya Solak1
6 affiliations
  1. 1Department of Dermatology, Faculty of Medicine, Trakya University, Trakya University Balkan Campus, Iskender, Edirne, Türkiye, 90 5054511134.
  2. 2Department of Dermatology, Istanbul Training and Research Hospital, Istanbul, Türkiye.
  3. 3Department of Dermatology, Etlik City Hospital, Ankara, Türkiye.
  4. 4Department of Dermatology, Medeniyet University Faculty of Medicine, Göztepe Prof. Dr. Suleyman Yalcin City Hospital, Istanbul, Türkiye.
  5. 5Department of Dermatology, School of Medicine, Acıbadem Mehmet Ali Aydınlar University, Istanbul, Türkiye.
  6. 6Department of Biostatistics and Medical Informatics, Trakya University Faculty of Medicine, Edirne, Türkiye.
Read full textPDFDOICite

Abstract

Background

The use of AI in health care is rapidly expanding due to its accessibility, with growing reliance among both physicians and patients.Objective

This study evaluated ChatGPT (OpenAI) and Gemini (Google) responses to patient questions on systemic isotretinoin.Methods

A total of 40 commonly anticipated patient questions were developed based on online sources and patient education materials and categorized into 5 groups. Responses were generated using the January 2025 free versions of ChatGPT (GPT-4o) and Gemini 2.0 Flash. A total of 5 experienced dermatologists evaluated the responses in a blinded manner using a 5-point Likert scale assessing accuracy, clarity, completeness, usefulness, and overall quality score (Global Quality Score [GQS]). Readability was measured using Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease, and Simple Measure of Gobbledygook (SMOG) index scores. Mixed-effects modeling was performed to adjust for reviewer and question variability, and intraclass correlation coefficients (ICCs) were calculated to assess interrater reliability.Results

ChatGPT achieved significantly higher ratings in accuracy, completeness, usefulness, and GQS, while clarity showed a nonsignificant trend. When questions were analyzed by subtype, no statistically significant differences were observed between models across evaluation parameters. Mixed-effects modeling confirmed higher odds of superior ratings for ChatGPT across domains (odds ratios [ORs] 1.82-5.88; all P<.001). However, interrater reliability was lower for ChatGPT (global ICC 0.401) compared with Gemini (global ICC 0.677). Readability analyses demonstrated that ChatGPT required a higher reading level (FKGL 12.6 vs 11.2; SMOG 11.3 vs 10.0).Conclusions

ChatGPT outperformed Gemini in terms of data accuracy and quality, while Gemini offered superior readability and consistency. Both models exceeded recommended readability thresholds, highlighting a gap between informational quality and linguistic accessibility in AI-generated health communication.

Related

  1. Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
    Diagnostics (Basel, Switzerland)Peer-reviewedCC BYFull textGeneral medicineEurope PMC
  2. Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
    Çağlasu Sancaktar, Burak Çakmak
    Digital HealthPeer-reviewedCC BYFull textEpidemiology & public healthDOAJ
  3. Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk.
    Emre Ulubaş, Damla Maide Davutluoğlu, İnci Rana Karaca +1
    BMC oral healthPeer-reviewedCC BY-NC-NDFull textEpidemiology & public healthEurope PMCDOAJ
  4. How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
    Ahmet Kürşat Kara, Ozan Işık
    Frontiers in surgeryPeer-reviewedCC BYFull textSurgery & anaesthesiaEurope PMC
  5. Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
    Yeliz Yilmaz Bozok, Nihan Acar, Murat Kemal Atahan +1
    PLOS OnePeer-reviewedCC BY 4.0Full textOncologyPLOS

Cite

BibTeXRIS
BibTeX
@article{avcı2026quality,
  title = {{Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.}},
  author = {Elif Bal Avcı and Buğra Burç Dağtaş and Dilara İlhan Erdil and Ozan Erdem and Andaç Salman and Selçuk Korkmaz and Sezgi Sarıkaya Solak},
  journal = {JMIR dermatology},
  year = {2026},
  month = oct,
  doi = {10.2196/99780},
  url = {https://europepmc.org/article/MED/42961658},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study.
AU  - Elif Bal Avcı
AU  - Buğra Burç Dağtaş
AU  - Dilara İlhan Erdil
AU  - Ozan Erdem
AU  - Andaç Salman
AU  - Selçuk Korkmaz
AU  - Sezgi Sarıkaya Solak
PY  - 2026
DA  - 2026/10/30
JO  - JMIR dermatology
DO  - 10.2196/99780
UR  - https://europepmc.org/article/MED/42961658
AB  - Background  The use of AI in health care is rapidly expanding due to its accessibility, with growing reliance among both physicians and patients.Objective  This study evaluated ChatGPT (OpenAI) and Gemini (Google) responses to patient questions on systemic isotretinoin.Methods  A total of 40 commonly anticipated patient questions were developed based on online sources and patient education materials and categorized into 5 groups. Responses were generated using the January 2025 free versions of ChatGPT (GPT-4o) and Gemini 2.0 Flash. A total of 5 experienced dermatologists evaluated the responses in a blinded manner using a 5-point Likert scale assessing accuracy, clarity, completeness, usefulness, and overall quality score (Global Quality Score [GQS]). Readability was measured using Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease, and Simple Measure of Gobbledygook (SMOG) index scores. Mixed-effects modeling was performed to adjust for reviewer and question variability, and intraclass correlation coefficients (ICCs) were calculated to assess interrater reliability.Results  ChatGPT achieved significantly higher ratings in accuracy, completeness, usefulness, and GQS, while clarity showed a nonsignificant trend. When questions were analyzed by subtype, no statistically significant differences were observed between models across evaluation parameters. Mixed-effects modeling confirmed higher odds of superior ratings for ChatGPT across domains (odds ratios [ORs] 1.82-5.88; all P<.001). However, interrater reliability was lower for ChatGPT (global ICC 0.401) compared with Gemini (global ICC 0.677). Readability analyses demonstrated that ChatGPT required a higher reading level (FKGL 12.6 vs 11.2; SMOG 11.3 vs 10.0).Conclusions  ChatGPT outperformed Gemini in terms of data accuracy and quality, while Gemini offered superior readability and consistency. Both models exceeded recommended readability thresholds, highlighting a gap between informational quality and linguistic accessibility in AI-generated health communication.
SN  - 2562-0959
ER  -