Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models.
5 affiliations
- 1The University of Kansas School of Medicine-Kansas City, Kansas City, KS.
- 2Department of CardiovascularMedicine.
- 3Department of Biostatistics and Data Science.
- 4Department of Medical Informatics.
- 5Department of Electrical Engineering and Computer Science, The University of Kansas, Lawrence, KS, USA.
Abstract
Introduction
Large language models (LLMs) are being explored for multiple applications in medical research, including medical text classification. Authors of the study evaluated the performance of three off-the-shelf LLMs in classifying unstructured free-text CT pulmonary angiography reports for pulmonary embolism (PE)-related diagnostic labels.Methods
Authors assessed human-labeled CT reports using three LLMs (ChatGPT-4o, Llama 3.3 70B, and Llama 3.1 8B) with zero-shot simple and chain-of-thought (CoT) prompts. Classification tasks included acute PE (N = 1,025) and, among reports with PE (n = 660), bilateral PE and large PE.Results
LLMs demonstrated high classification performance for acute PE and bilateral PE but lower performance for large PE. With simple prompts, the highest F1 scores were 97.0% for acute PE (Llama 3.1 8B; sensitivity, 95.6%; specificity, 97.3%), 92.9% for bilateral PE (ChatGPT-4o; sensitivity, 98.0%; specificity, 88.6%), and 72.8% for large PE (ChatGPT-4o; sensitivity, 82.9%; specificity, 82.2%). CoT prompting did not meaningfully improve acute PE classification but provided slight improvements in subclassification performance for bilateral and large PE.Conclusions
Off-the-shelf LLMs can classify free-text CT reports using simple prompt engineering and may facilitate automated data extraction for medical research following appropriate evaluation of classification performance.
Related
- Automated Interpretation of EEG Reports Using a Large Language Model with Structured Confidence Outputs
- Large language model-assisted CAD-RADS classification: structured report extraction, accuracy, and clinical decision-making risk of ChatGPT-4o/5 and DeepSeek-R1
- A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo
- Comparative analysis of open-source locally deployed LLMs for extraction of diagnostic information from clinical CMR imaging reports
- LLM-Based Extraction of Clinical Practice Guidelines into Structured Arguments
BibTeX
@article{gupta2026zeroshot,
title = {{Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models.}},
author = {Amulya Gupta and Alex Hongslo and Quynh Nguyen and Jake Caldwell and Ben Choi and Christopher J Harvey and Jeffrey Thompson and Diego Mazzotti and Zijun Yao and Amit Noheria},
journal = {Kansas journal of medicine},
year = {2026},
month = sep,
doi = {10.17161/kjm.vol19.25396},
url = {https://europepmc.org/article/MED/42921394},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models.
AU - Amulya Gupta
AU - Alex Hongslo
AU - Quynh Nguyen
AU - Jake Caldwell
AU - Ben Choi
AU - Christopher J Harvey
AU - Jeffrey Thompson
AU - Diego Mazzotti
AU - Zijun Yao
AU - Amit Noheria
PY - 2026
DA - 2026/09/01
JO - Kansas journal of medicine
DO - 10.17161/kjm.vol19.25396
UR - https://europepmc.org/article/MED/42921394
AB - Introduction Large language models (LLMs) are being explored for multiple applications in medical research, including medical text classification. Authors of the study evaluated the performance of three off-the-shelf LLMs in classifying unstructured free-text CT pulmonary angiography reports for pulmonary embolism (PE)-related diagnostic labels.Methods Authors assessed human-labeled CT reports using three LLMs (ChatGPT-4o, Llama 3.3 70B, and Llama 3.1 8B) with zero-shot simple and chain-of-thought (CoT) prompts. Classification tasks included acute PE (N = 1,025) and, among reports with PE (n = 660), bilateral PE and large PE.Results LLMs demonstrated high classification performance for acute PE and bilateral PE but lower performance for large PE. With simple prompts, the highest F1 scores were 97.0% for acute PE (Llama 3.1 8B; sensitivity, 95.6%; specificity, 97.3%), 92.9% for bilateral PE (ChatGPT-4o; sensitivity, 98.0%; specificity, 88.6%), and 72.8% for large PE (ChatGPT-4o; sensitivity, 82.9%; specificity, 82.2%). CoT prompting did not meaningfully improve acute PE classification but provided slight improvements in subclassification performance for bilateral and large PE.Conclusions Off-the-shelf LLMs can classify free-text CT reports using simple prompt engineering and may facilitate automated data extraction for medical research following appropriate evaluation of classification performance.
SN - 1948-2035
ER -