Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis
- 1Radboudumc
Abstract
In this study, LLM and prompting strategies for labeling Specific Incidental Findings (SIFs) occurring during lung cancer screening, in radiology reports were compared. In this retrospective study, 300 CT reports from a Dutch academic hospital were included between 19th March 2003 and 28th December 2020. GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, a keyword-based search method, and a radiologist labelled the reports for the presence or absence of nine SIFs. The radiologist's scores were the reference standard. Each LLM was evaluated with four different prompting strategies: two without Chain-of-Thought (CoT), and two with CoT (one zero shot, one few shot). The automated labelling methods were evaluated against the radiologist primarily using Cohen's Kappa, and additionally sensitivity, specificity, and F1 scores. GPT-5 achieved substantial or almost perfect agreement for seven out of nine SIFs: aortic valve disease, bronchiectasis, consolidation, coronary artery calcification, emphysema, abnormality in mediastinal lymph nodes, and thyroid abnormalities, with Cohen κ values of 0.81 (0.70, 0.93), 0.97 (0.93, 1.00), 0.71 (0.62, 0.82), 0.92 (0.87, 0.96), 0.91 (0.84, 0.97), 0.71 (0.62, 0.80), and 0.92 (0.86, 0.98) respectively. For the remaining two SIFs, moderate LLM agreement was obtained with MedGemma-4B-IT and with GPT-4-Turbo with Cohen κ values of 0.48 (0.40, 0.60), 0.42 (0.31, 0.58) for interstitial lung abnormalities and mediastinal mass respectively. The highest agreement was obtained with the non-CoT prompt strategies for most SIFs. This study demonstrated that LLMs can automatically label radiology reports with agreement ranging from almost perfect to moderate, but not one LLM -- prompt combination achieved the highest agreement across all SIFs.
Related
- Enhancing medical education with AI: A comparative study of various large language models in radiology
- Comparative efficacy of large language models (Google Gemini and ChatGPT 4.0) in screening patients for diabetic retinopathy
- CO003 | GENOMIC INCIDENTAL FINDINGS IN DISORDERS OF HEMOSTASIS
- Sycophancy in Large Language Models in Oral and Maxillofacial Radiology: The Impact of Misleading Context on Image- and Text-Based Diagnostic Scenarios
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
BibTeX
@misc{graaf2026large,
title = {{Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis}},
author = {F. van der Graaf and N. Antonissen and E. T. Scholten and H. Schalkx and M. Prokop and C. Jacobs},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.09.26367611},
url = {https://www.medrxiv.org/content/10.64898/2026.11.09.26367611v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis
AU - F. van der Graaf
AU - N. Antonissen
AU - E. T. Scholten
AU - H. Schalkx
AU - M. Prokop
AU - C. Jacobs
PY - 2026
DA - 2026/11/09
PB - medRxiv
DO - 10.64898/2026.11.09.26367611
UR - https://www.medrxiv.org/content/10.64898/2026.11.09.26367611v1
AB - In this study, LLM and prompting strategies for labeling Specific Incidental Findings (SIFs) occurring during lung cancer screening, in radiology reports were compared. In this retrospective study, 300 CT reports from a Dutch academic hospital were included between 19th March 2003 and 28th December 2020. GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, a keyword-based search method, and a radiologist labelled the reports for the presence or absence of nine SIFs. The radiologist's scores were the reference standard. Each LLM was evaluated with four different prompting strategies: two without Chain-of-Thought (CoT), and two with CoT (one zero shot, one few shot). The automated labelling methods were evaluated against the radiologist primarily using Cohen's Kappa, and additionally sensitivity, specificity, and F1 scores. GPT-5 achieved substantial or almost perfect agreement for seven out of nine SIFs: aortic valve disease, bronchiectasis, consolidation, coronary artery calcification, emphysema, abnormality in mediastinal lymph nodes, and thyroid abnormalities, with Cohen κ values of 0.81 (0.70, 0.93), 0.97 (0.93, 1.00), 0.71 (0.62, 0.82), 0.92 (0.87, 0.96), 0.91 (0.84, 0.97), 0.71 (0.62, 0.80), and 0.92 (0.86, 0.98) respectively. For the remaining two SIFs, moderate LLM agreement was obtained with MedGemma-4B-IT and with GPT-4-Turbo with Cohen κ values of 0.48 (0.40, 0.60), 0.42 (0.31, 0.58) for interstitial lung abnormalities and mediastinal mass respectively. The highest agreement was obtained with the non-CoT prompt strategies for most SIFs. This study demonstrated that LLMs can automatically label radiology reports with agreement ranging from almost perfect to moderate, but not one LLM -- prompt combination achieved the highest agreement across all SIFs.
ER -