Generative AI and Clinicians Show Comparable Prognostic Reasoning From Clinical Narratives in Biologic‐Treated CRSwNP
8 affiliations
- 1City St. George's University of London, School of Medicine Program delivered by University of Nicosia at the Chaim Sheba Medical Center Ramat Gan Israel
- 2Department of Otolaryngology‐Head and Neck Surgery, Integrated Surgical Institute Cleveland Clinic Foundation Cleveland Ohio USA
- 3Department of Otolaryngology Valencia University General Hospital Valencia Spain
- 4Department of Otolaryngology‐Head and Neck Surgery Meir Medical Center Kfar Saba Israel
- 5Hospices civils de Lyon, Hôpital Lyon Sud, Service d'ORL, d'otoneurochirurgie et de chirurgie cervico‐faciale Pierre Bénite France
- 6Department of Otolaryngology Sheba Medical Center Ramat Gan Israel
- 7Department of Otolaryngology‐Head and Neck Surgery Houston Methodist Hospital Houston Texas USA
- 8Division of Otolaryngology, Department of Surgery Cedars‐Sinai Medical Center Los Angeles California USA
Abstract
Objective To compare the ability of large language models (LLMs) and otolaryngologists to identify prognostic signals from brief clinical vignettes in CRSwNP. Methods In this blinded study, 68 adults initiating biologic therapy for CRSwNP (≥ 36 months follow‐up) were represented by standardized vignettes derived from documentation immediately before biologic initiation. Vignettes summarized symptoms, prior surgery, comorbidities, and medications. Biomarkers, imaging scores, smell testing, and follow‐up data were excluded to isolate text‐based prognostic reasoning under identical constraints. Five attending otolaryngologists, one rhinology fellow, and two residents independently predicted four 5‐year outcomes: subsequent endoscopic sinus surgery, recurrent systemic steroid bursts, biologic switch, and a composite endpoint. Multiple LLMs were queried in identical zero‐shot format across three sessions to assess stability. Predictions were compared with verified outcomes using accuracy, sensitivity, specificity, F1 score, Cohen's κ, and Matthews correlation coefficient. Results Five‐year outcome prevalences were 38.2% (26 of 68) for subsequent sinus surgery, 26.5% (18 of 68) for steroid bursts, 14.7% (10 of 68) for biologic switch, and 58.8% (40 of 68) for composite failure. Macro‐averaged LLM accuracies ranged from 65.0% to 76.1%, compared with 65.7% ± 7.4% for human raters. The best‐performing LLM achieved accuracy comparable to the top attending. Both groups showed higher negative than positive predictive values, indicating better discrimination for patients without adverse events. Agreement between LLMs and clinicians was moderate and within the range of inter‐clinician variability. Conclusion When restricted to short pretreatment text, LLMs produced prognostic judgments within the range of clinician variability for severe, biologic‐treated CRSwNP outcomes. These findings suggest that in information‐limited settings, LLMs may provide an auxiliary reasoning signal rather than a standalone decision tool.
Related
- A Pragmatic Randomized Trial of an EHR-Integrated Generative AI Chart Summarization Tool for Ambulatory Clinicians
- Reclaiming human clinical reasoning in generative AI-enabled healthcare: a perspective on cognitive agency, skill evolution, and accountability
- Leveraging prompt-driven generative AI for systematic reviews in digital psychiatry: A stage-matched comparative proof-of-concept for healthcare researchers and clinicians
- Effectiveness of Generative AI–Enabled Teaching for Clinical Reasoning in Health Care Education: Protocol for a Systematic Review and Meta-Analysis
- Retrospective Narratives of Childhood and Adolescent Trauma Among Help-Seeking Adult men with High ACE Exposure
BibTeX
@article{hack2026generative,
title = {{Generative AI and Clinicians Show Comparable Prognostic Reasoning From Clinical Narratives in Biologic‐Treated CRSwNP}},
author = {Sholem Hack and Chase Kahn and Ainhoa Garcia‐Lliberos and Cristina Rodriguez‐Prado and Ameen Biadsee and Maxime Fieux and Mark Liu and Eran Glikson and Masayoshi Takashima and Omar G. Ahmed and Raj Sindwani and Dennis M. Tang and Christopher R. Roxbury},
journal = {World Journal of Otorhinolaryngology-Head and Neck Surgery},
year = {2026},
month = oct,
doi = {10.1002/wjo2.70164},
url = {https://doi.org/10.1002/wjo2.70164},
note = {CC BY},
}
RIS
TY - JOUR
TI - Generative AI and Clinicians Show Comparable Prognostic Reasoning From Clinical Narratives in Biologic‐Treated CRSwNP
AU - Sholem Hack
AU - Chase Kahn
AU - Ainhoa Garcia‐Lliberos
AU - Cristina Rodriguez‐Prado
AU - Ameen Biadsee
AU - Maxime Fieux
AU - Mark Liu
AU - Eran Glikson
AU - Masayoshi Takashima
AU - Omar G. Ahmed
AU - Raj Sindwani
AU - Dennis M. Tang
AU - Christopher R. Roxbury
PY - 2026
DA - 2026/10/23
JO - World Journal of Otorhinolaryngology-Head and Neck Surgery
DO - 10.1002/wjo2.70164
UR - https://doi.org/10.1002/wjo2.70164
AB - Objective To compare the ability of large language models (LLMs) and otolaryngologists to identify prognostic signals from brief clinical vignettes in CRSwNP. Methods In this blinded study, 68 adults initiating biologic therapy for CRSwNP (≥ 36 months follow‐up) were represented by standardized vignettes derived from documentation immediately before biologic initiation. Vignettes summarized symptoms, prior surgery, comorbidities, and medications. Biomarkers, imaging scores, smell testing, and follow‐up data were excluded to isolate text‐based prognostic reasoning under identical constraints. Five attending otolaryngologists, one rhinology fellow, and two residents independently predicted four 5‐year outcomes: subsequent endoscopic sinus surgery, recurrent systemic steroid bursts, biologic switch, and a composite endpoint. Multiple LLMs were queried in identical zero‐shot format across three sessions to assess stability. Predictions were compared with verified outcomes using accuracy, sensitivity, specificity, F1 score, Cohen's κ, and Matthews correlation coefficient. Results Five‐year outcome prevalences were 38.2% (26 of 68) for subsequent sinus surgery, 26.5% (18 of 68) for steroid bursts, 14.7% (10 of 68) for biologic switch, and 58.8% (40 of 68) for composite failure. Macro‐averaged LLM accuracies ranged from 65.0% to 76.1%, compared with 65.7% ± 7.4% for human raters. The best‐performing LLM achieved accuracy comparable to the top attending. Both groups showed higher negative than positive predictive values, indicating better discrimination for patients without adverse events. Agreement between LLMs and clinicians was moderate and within the range of inter‐clinician variability. Conclusion When restricted to short pretreatment text, LLMs produced prognostic judgments within the range of clinician variability for severe, biologic‐treated CRSwNP outcomes. These findings suggest that in information‐limited settings, LLMs may provide an auxiliary reasoning signal rather than a standalone decision tool.
SN - 2589-1081
SN - 2095-8811
ER -