Assessing the reliability of AI-generated responses to patient-oriented lung cancer questions: a multi-platform study.
- 1Department of Thoracic Surgery, Bursa Uludağ University Faculty of Medicine, 1st Floor, 16059, Bursa, Turkey. [email protected].
- 2Department of Thoracic Surgery, Afyon State Hospital, Afyon, Turkey.
Abstract
Background
Large language models (LLMs) are increasingly used by patients seeking lung cancer-related medical information. Recent studies have increasingly compared multiple LLMs for patient-facing medical information; however, differences in clinically relevant performance dimensions and consistency across repeated responses remain important areas for evaluation.Methods
Seventy-nine lung cancer-related patient questions were queried across three freely accessible LLM platforms (ChatGPT, Claude, and Google Gemini) between December 2025 and January 2026. Responses were independently evaluated by two specialist thoracic surgeons for accuracy, completeness, clinical relevance, and readability, with disagreements resolved by consensus. Reproducibility was assessed through five repeated independent queries using a clinical-equivalence framework, and automated readability was assessed using the Flesch-Kincaid Grade Level. Ordinal outcomes were analyzed using cumulative link mixed models accounting for repeated evaluation of questions, while reproducibility was analyzed using generalized estimating equations.Results
All platforms demonstrated high accuracy, with no significant overall platform difference (p = 0.150). Completeness differed significantly across platforms (p < 0.001), with higher ratings for ChatGPT than Claude and Gemini and for Claude than Gemini. Clinical relevance showed no overall platform effect (p = 0.568), although a platform-by-domain interaction was observed (p = 0.046). Expert-rated readability differed significantly (p < 0.001), with Gemini receiving higher ratings than ChatGPT and Claude. All platforms produced content requiring approximately 10th-12th grade reading levels. Reproducibility was high overall (88.6-96.2%), with no statistically significant overall platform effect (p = 0.072).Conclusions
Contemporary LLMs demonstrated high accuracy and clinical relevance, but differed in completeness and readability. Reproducibility was high across platforms, while all models exceeded recommended patient reading levels. Further patient-based validation is needed before routine clinical integration.Clinical trial number
Not applicable.
Related
- Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
- Comparative evaluation of ChatGPT and gemini responses to patient-oriented questions on breast cancer
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
BibTeX
@article{ozer2026assessing,
title = {{Assessing the reliability of AI-generated responses to patient-oriented lung cancer questions: a multi-platform study.}},
author = {Erhan Özer and Gizem Gedikoğlu Pirim},
journal = {BMC pulmonary medicine},
year = {2026},
month = oct,
doi = {10.1186/s12890-026-04903-x},
url = {https://europepmc.org/article/MED/42972341},
note = {CC BY},
}
RIS
TY - JOUR
TI - Assessing the reliability of AI-generated responses to patient-oriented lung cancer questions: a multi-platform study.
AU - Erhan Özer
AU - Gizem Gedikoğlu Pirim
PY - 2026
DA - 2026/10/23
JO - BMC pulmonary medicine
DO - 10.1186/s12890-026-04903-x
UR - https://europepmc.org/article/MED/42972341
AB - Background Large language models (LLMs) are increasingly used by patients seeking lung cancer-related medical information. Recent studies have increasingly compared multiple LLMs for patient-facing medical information; however, differences in clinically relevant performance dimensions and consistency across repeated responses remain important areas for evaluation.Methods Seventy-nine lung cancer-related patient questions were queried across three freely accessible LLM platforms (ChatGPT, Claude, and Google Gemini) between December 2025 and January 2026. Responses were independently evaluated by two specialist thoracic surgeons for accuracy, completeness, clinical relevance, and readability, with disagreements resolved by consensus. Reproducibility was assessed through five repeated independent queries using a clinical-equivalence framework, and automated readability was assessed using the Flesch-Kincaid Grade Level. Ordinal outcomes were analyzed using cumulative link mixed models accounting for repeated evaluation of questions, while reproducibility was analyzed using generalized estimating equations.Results All platforms demonstrated high accuracy, with no significant overall platform difference (p = 0.150). Completeness differed significantly across platforms (p < 0.001), with higher ratings for ChatGPT than Claude and Gemini and for Claude than Gemini. Clinical relevance showed no overall platform effect (p = 0.568), although a platform-by-domain interaction was observed (p = 0.046). Expert-rated readability differed significantly (p < 0.001), with Gemini receiving higher ratings than ChatGPT and Claude. All platforms produced content requiring approximately 10th-12th grade reading levels. Reproducibility was high overall (88.6-96.2%), with no statistically significant overall platform effect (p = 0.072).Conclusions Contemporary LLMs demonstrated high accuracy and clinical relevance, but differed in completeness and readability. Reproducibility was high across platforms, while all models exceeded recommended patient reading levels. Further patient-based validation is needed before routine clinical integration.Clinical trial number Not applicable.
SN - 1471-2466
ER -