Large language models for pneumonia detection in radiology reports via text analysis
5 affiliations
- 1Department of Intensive Care Unit, The First Affiliated Hospital of Jinan University, Guangzhou, China
- 2School of Nursing, Jinan University, Guangzhou, China
- 3Department of Critical Care Medicine, Shenzhen People’s Hospital, The Second Clinical Medical College, Jinan University, Shenzhen, China
- 4Department of Pharmacy, The First Affiliated Hospital of Jinan University, Guangzhou, China
- 5Clinical Research Center, The First Affiliated Hospital of Jinan University, Guangzhou, China
Abstract
Pneumonia is a common infection in critically ill patients with poor outcomes, and its identification relies heavily on radiological evidence. Most MIMIC-based pneumonia studies rely on structured codes rather than radiological text, which may limit the precision of pneumonia-related case identification. This study evaluated large language models (LLMs) and retrieval-augmented generation (RAG) frameworks for automated detection of radiographic evidence of pneumonia from radiology reports. Radiological reports from the MIMIC-IV database were analyzed using pre-trained models in an inference-only setting without additional training, fine-tuning, or dataset splitting. Model performance was evaluated using accuracy, precision, sensitivity, specificity, and F1-score, with point estimates calculated from the full dataset and corresponding 95% confidence intervals estimated using 1,000 patient-level clustered bootstrap replicates. Independent evaluation was subsequently performed using a GZ dataset from a tertiary hospital in Guangzhou. RAG generally improved model performance, with gemma3-27b achieving the highest accuracy of 96.4% on the MIMIC-IV dataset and maintaining an accuracy of 87.6% on the GZ dataset. Model performance did not increase linearly with parameter size, suggesting that task-specific model suitability may be more important than parameter scale for this application. Although larger models required longer inference times, their increased computational costs did not consistently translate into proportional performance gains. Overall, LLMs demonstrated high accuracy in interpreting radiology reports for detecting radiographic evidence of pneumonia. The RAG architecture further enhanced robustness and generalizability through knowledge augmentation, providing a methodological basis for automated radiology report analysis and large-scale database curation. Author summary: In this study, we explored whether modern artificial intelligence tools can help identify radiographic evidence of pneumonia more accurately by reading radiology reports. This condition is a frequent and serious infection in intensive care units, and its diagnosis often depends on how clinicians interpret imaging findings. Many previous studies have relied on coded data, which may miss important details contained in free-text reports. We used large language models and a retrieval-based approach to analyze radiology reports and determine whether pneumonia was present. We found that these models can achieve high accuracy and remain reliable when applied to data from a different hospital, suggesting that the approach is broadly applicable. Importantly, we observed that simply using larger and more computationally expensive models does not always lead to better results. Instead, models that are better aligned with the specific clinical task tend to perform best. Our findings suggest that artificial intelligence can support more precise and scalable identification of radiographic evidence of pneumonia from routine clinical data. This may help improve research quality and support future clinical decision-support applications.
Related
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study
- Evaluating large language models for lay summaries of radiology reports using tailored prompting strategies and mixed-method assessment
- Natural Language Processing Based Solution for Labeling Brain Metastasis Identified in Radiology Reports
- Sycophancy in Large Language Models in Oral and Maxillofacial Radiology: The Impact of Misleading Context on Image- and Text-Based Diagnostic Scenarios
BibTeX
@article{chen2026large,
title = {{Large language models for pneumonia detection in radiology reports via text analysis}},
author = {Weisheng Chen and Xinya Li and Zhongkai Qu and Ting Liu and Bo Chen and Liying Huang and Ningxia He and Zhigang Wang and Jun Lyu},
journal = {PLOS Digital Health},
year = {2026},
month = sep,
doi = {10.1371/journal.pdig.0001104},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001104},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Large language models for pneumonia detection in radiology reports via text analysis
AU - Weisheng Chen
AU - Xinya Li
AU - Zhongkai Qu
AU - Ting Liu
AU - Bo Chen
AU - Liying Huang
AU - Ningxia He
AU - Zhigang Wang
AU - Jun Lyu
PY - 2026
DA - 2026/09/23
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001104
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001104
AB - Pneumonia is a common infection in critically ill patients with poor outcomes, and its identification relies heavily on radiological evidence. Most MIMIC-based pneumonia studies rely on structured codes rather than radiological text, which may limit the precision of pneumonia-related case identification. This study evaluated large language models (LLMs) and retrieval-augmented generation (RAG) frameworks for automated detection of radiographic evidence of pneumonia from radiology reports. Radiological reports from the MIMIC-IV database were analyzed using pre-trained models in an inference-only setting without additional training, fine-tuning, or dataset splitting. Model performance was evaluated using accuracy, precision, sensitivity, specificity, and F1-score, with point estimates calculated from the full dataset and corresponding 95% confidence intervals estimated using 1,000 patient-level clustered bootstrap replicates. Independent evaluation was subsequently performed using a GZ dataset from a tertiary hospital in Guangzhou. RAG generally improved model performance, with gemma3-27b achieving the highest accuracy of 96.4% on the MIMIC-IV dataset and maintaining an accuracy of 87.6% on the GZ dataset. Model performance did not increase linearly with parameter size, suggesting that task-specific model suitability may be more important than parameter scale for this application. Although larger models required longer inference times, their increased computational costs did not consistently translate into proportional performance gains. Overall, LLMs demonstrated high accuracy in interpreting radiology reports for detecting radiographic evidence of pneumonia. The RAG architecture further enhanced robustness and generalizability through knowledge augmentation, providing a methodological basis for automated radiology report analysis and large-scale database curation. Author summary: In this study, we explored whether modern artificial intelligence tools can help identify radiographic evidence of pneumonia more accurately by reading radiology reports. This condition is a frequent and serious infection in intensive care units, and its diagnosis often depends on how clinicians interpret imaging findings. Many previous studies have relied on coded data, which may miss important details contained in free-text reports. We used large language models and a retrieval-based approach to analyze radiology reports and determine whether pneumonia was present. We found that these models can achieve high accuracy and remain reliable when applied to data from a different hospital, suggesting that the approach is broadly applicable. Importantly, we observed that simply using larger and more computationally expensive models does not always lead to better results. Instead, models that are better aligned with the specific clinical task tend to perform best. Our findings suggest that artificial intelligence can support more precise and scalable identification of radiographic evidence of pneumonia from routine clinical data. This may help improve research quality and support future clinical decision-support applications.
SN - 2767-3170
ER -