General-purpose and medically adapted large language models versus oncologist consensus for immunotherapy videos
Abstract
IntroductionPatients receiving immune checkpoint inhibitors depend on accurate education to recognize immune-related adverse events and to know when to contact their oncology team, and many rely on online video of highly variable quality. Expert review of such material is resource-intensive, and large language models (LLMs) are increasingly proposed to support preliminary screening; whether any model agrees with specialist judgment closely enough to do so safely, and how often models pass expert-inadequate material as adequate, is unknown.MethodsIn this cross-sectional content analysis (reported per STROBE), we evaluated anonymized transcripts of 90 English-language YouTube videos on cancer immunotherapy. Three medical oncologists and four LLMs — ChatGPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and the medically adapted MedGemma 27B — independently scored every transcript using DISCERN, the Global Quality Score (GQS), and the Patient Education Materials Assessment Tool (PEMAT) understandability and actionability domains, each model in three separate runs, with expert majority consensus as the reference standard.ResultsAgreement among oncologists was good to excellent, whereas agreement among models was limited. By expert-majority classification, 35 of 90 transcripts (38.9%) met at least three of the four adequacy criteria and 24 (26.7%) met all four. ChatGPT-5.5 deviated least from expert scores on every instrument and classified adequacy most accurately (sensitivity 94.3%, 95% CI 81.4–98.4; specificity 89.1%, 78.2–94.9; Cohen’s kappa 0.82, 0.69–0.93). Gemini and MedGemma scored systematically more leniently under the tested conditions and produced 43 and 26 false-positive adequacy classifications, respectively, passing material the oncologists judged inadequate; the medically adapted model did not improve agreement in this evaluation, and raw popularity metrics did not identify high-quality content.DiscussionThe transcript-based educational content of most sampled videos did not meet all four adequacy criteria. These findings support further evaluation of selected model–prompt combinations for preliminary review of patient-education material, but they do not establish an operating threshold for clinical use, and the leniency observed in two models means expert review cannot be removed from the process.
Related
- Cross-corpus consensus feature selection for language-invariant Parkinson’s disease detection: a dual-stream acoustic and spectrogram-attention framework
- Validation of the Amharic Version of the Full and Short Form of the Trust in Oncologist Scale Among Cervical Cancer Patient
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- No significant differences between large language model prompting strategies and human experts in summarisation of 'the formal EU-US meniscus rehabilitation consensus: An ESSKA-AOSSM-AASPT Initiative'.
BibTeX
@article{yaslikaya2026generalpurpose,
title = {{General-purpose and medically adapted large language models versus oncologist consensus for immunotherapy videos}},
author = {Şendağ Yaslikaya and Şuheda Ataş İpek and Esra Erdem Palaz},
journal = {Frontiers in Public Health},
year = {2026},
month = oct,
doi = {10.3389/fpubh.2026.1962857},
url = {https://www.frontiersin.org/articles/10.3389/fpubh.2026.1962857},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - General-purpose and medically adapted large language models versus oncologist consensus for immunotherapy videos
AU - Şendağ Yaslikaya
AU - Şuheda Ataş İpek
AU - Esra Erdem Palaz
PY - 2026
DA - 2026/10/30
JO - Frontiers in Public Health
DO - 10.3389/fpubh.2026.1962857
UR - https://www.frontiersin.org/articles/10.3389/fpubh.2026.1962857
AB - IntroductionPatients receiving immune checkpoint inhibitors depend on accurate education to recognize immune-related adverse events and to know when to contact their oncology team, and many rely on online video of highly variable quality. Expert review of such material is resource-intensive, and large language models (LLMs) are increasingly proposed to support preliminary screening; whether any model agrees with specialist judgment closely enough to do so safely, and how often models pass expert-inadequate material as adequate, is unknown.MethodsIn this cross-sectional content analysis (reported per STROBE), we evaluated anonymized transcripts of 90 English-language YouTube videos on cancer immunotherapy. Three medical oncologists and four LLMs — ChatGPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and the medically adapted MedGemma 27B — independently scored every transcript using DISCERN, the Global Quality Score (GQS), and the Patient Education Materials Assessment Tool (PEMAT) understandability and actionability domains, each model in three separate runs, with expert majority consensus as the reference standard.ResultsAgreement among oncologists was good to excellent, whereas agreement among models was limited. By expert-majority classification, 35 of 90 transcripts (38.9%) met at least three of the four adequacy criteria and 24 (26.7%) met all four. ChatGPT-5.5 deviated least from expert scores on every instrument and classified adequacy most accurately (sensitivity 94.3%, 95% CI 81.4–98.4; specificity 89.1%, 78.2–94.9; Cohen’s kappa 0.82, 0.69–0.93). Gemini and MedGemma scored systematically more leniently under the tested conditions and produced 43 and 26 false-positive adequacy classifications, respectively, passing material the oncologists judged inadequate; the medically adapted model did not improve agreement in this evaluation, and raw popularity metrics did not identify high-quality content.DiscussionThe transcript-based educational content of most sampled videos did not meet all four adequacy criteria. These findings support further evaluation of selected model–prompt combinations for preliminary review of patient-education material, but they do not establish an operating threshold for clinical use, and the leniency observed in two models means expert review cannot be removed from the process.
SN - 2296-2565
ER -