{ "abstract": "Pneumonia is a common infection in critically ill patients with poor outcomes, and its identification relies heavily on radiological evidence. Most MIMIC-based pneumonia studies rely on structured codes rather than radiological text, which may limit the precision of pneumonia-related case identification. This study evaluated large language models (LLMs) and retrieval-augmented generation (RAG) frameworks for automated detection of radiographic evidence of pneumonia from radiology reports. Radiological reports from the MIMIC-IV database were analyzed using pre-trained models in an inference-only setting without additional training, fine-tuning, or dataset splitting. Model performance was evaluated using accuracy, precision, sensitivity, specificity, and F1-score, with point estimates calculated from the full dataset and corresponding 95% confidence intervals estimated using 1,000 patient-level clustered bootstrap replicates. Independent evaluation was subsequently performed using a GZ dataset from a tertiary hospital in Guangzhou. RAG generally improved model performance, with gemma3-27b achieving the highest accuracy of 96.4% on the MIMIC-IV dataset and maintaining an accuracy of 87.6% on the GZ dataset. Model performance did not increase linearly with parameter size, suggesting that task-specific model suitability may be more important than parameter scale for this application. Although larger models required longer inference times, their increased computational costs did not consistently translate into proportional performance gains. Overall, LLMs demonstrated high accuracy in interpreting radiology reports for detecting radiographic evidence of pneumonia. The RAG architecture further enhanced robustness and generalizability through knowledge augmentation, providing a methodological basis for automated radiology report analysis and large-scale database curation. Author summary: In this study, we explored whether modern artificial intelligence tools can help identify radiographic evidence of pneumonia more accurately by reading radiology reports. This condition is a frequent and serious infection in intensive care units, and its diagnosis often depends on how clinicians interpret imaging findings. Many previous studies have relied on coded data, which may miss important details contained in free-text reports. We used large language models and a retrieval-based approach to analyze radiology reports and determine whether pneumonia was present. We found that these models can achieve high accuracy and remain reliable when applied to data from a different hospital, suggesting that the approach is broadly applicable. Importantly, we observed that simply using larger and more computationally expensive models does not always lead to better results. Instead, models that are better aligned with the specific clinical task tend to perform best. Our findings suggest that artificial intelligence can support more precise and scalable identification of radiographic evidence of pneumonia from routine clinical data. This may help improve research quality and support future clinical decision-support applications.", "abstract_shown": true, "affiliations": [ "Department of Intensive Care Unit, The First Affiliated Hospital of Jinan University, Guangzhou, China", "School of Nursing, Jinan University, Guangzhou, China", "Department of Critical Care Medicine, Shenzhen People’s Hospital, The Second Clinical Medical College, Jinan University, Shenzhen, China", "Department of Pharmacy, The First Affiliated Hospital of Jinan University, Guangzhou, China", "Clinical Research Center, The First Affiliated Hospital of Jinan University, Guangzhou, China" ], "article_type": "Research Article", "arxiv": null, "author_total": 9, "authors": [ { "affs": [ 0 ], "name": "Weisheng Chen" }, { "affs": [ 1 ], "name": "Xinya Li" }, { "affs": [ 2 ], "name": "Zhongkai Qu" }, { "affs": [ 3 ], "name": "Ting Liu" }, { "affs": [ 4 ], "name": "Bo Chen" }, { "name": "Liying Huang" }, { "name": "Ningxia He" }, { "name": "Zhigang Wang" }, { "name": "Jun Lyu" } ], "bibtex": "https://research.draconitic.com/paper/0588gz08j56ad.bib", "cited_by": 0, "doi": "10.1371/journal.pdig.0001104", "fulltext_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001104", "id": "0588gz08j56ad", "issns": [ "2767-3170" ], "landing_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001104", "language": null, "licence": "CC-BY-4.0", "licence_label": "CC BY 4.0", "licence_source": "journal", "licence_url": "https://creativecommons.org/licenses/by/4.0/", "metadata_licence": null, "native_subjects": [ [ "openalex", "1702|Machine Learning in Healthcare" ], [ "plos", "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Language" ], [ "plos", "/Biology and life sciences/Neuroscience/Neuroimaging/Computed axial tomography" ], [ "plos", "/Biology and life sciences/Psychology/Cognitive psychology/Language" ], [ "plos", "/Medicine and health sciences/Diagnostic medicine/Diagnostic radiology" ], [ "plos", "/Medicine and health sciences/Diagnostic medicine/Diagnostic radiology/Tomography/Computed axial tomography" ], [ "plos", "/Medicine and health sciences/Health care/Health care facilities/Hospitals/Intensive care units" ], [ "plos", "/Medicine and health sciences/Medical conditions/Infectious diseases/Nosocomial infections" ], [ "plos", "/Medicine and health sciences/Medical conditions/Infectious diseases/Respiratory infections" ], [ "plos", "/Medicine and health sciences/Medical conditions/Respiratory disorders/Respiratory infections" ], [ "plos", "/Medicine and health sciences/Pulmonology/Pneumonia" ], [ "plos", "/Medicine and health sciences/Pulmonology/Respiratory disorders/Respiratory infections" ], [ "plos", "/Medicine and health sciences/Radiology and imaging" ], [ "plos", "/Medicine and health sciences/Radiology and imaging/Diagnostic radiology" ], [ "plos", "/Medicine and health sciences/Radiology and imaging/Diagnostic radiology/Tomography/Computed axial tomography" ], [ "plos", "/Research and analysis methods/Imaging techniques/Diagnostic radiology" ], [ "plos", "/Research and analysis methods/Imaging techniques/Diagnostic radiology/Tomography/Computed axial tomography" ], [ "plos", "/Research and analysis methods/Imaging techniques/Neuroimaging/Computed axial tomography" ], [ "plos", "/Social sciences/Psychology/Cognitive psychology/Language" ] ], "openalex": "W7214071438", "pdf_url": "https://journals.plos.org/digitalhealth/article/file?id=10.1371/journal.pdig.0001104&type=printable", "pmcid": null, "pmid": null, "published": "2026-09-23", "publisher": "Public Library of Science", "ris": "https://research.draconitic.com/paper/0588gz08j56ad.ris", "sources": [ { "first_seen": "2026-10-30T12:17:43Z", "native_id": "10.1371/journal.pdig.0001104", "source": "plos", "url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001104" } ], "subjects": [ { "field": "medicine", "origin": "plos:diagnostic radiology", "subfield": "imaging" } ], "title": "Large language models for pneumonia detection in radiology reports via text analysis", "title_text": "Large language models for pneumonia detection in radiology reports via text analysis", "type": "peer-reviewed", "updated_at": "2026-10-30T12:17:43Z", "url": "https://research.draconitic.com/paper/0588gz08j56ad", "venue": "PLOS Digital Health", "versions": [] }