Comparative analysis of open-source locally deployed LLMs for extraction of diagnostic information from clinical CMR imaging reports
Abstract
We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. We evaluated nine open-source LLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models. Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories. Google’s Gemma 2(27B) model achieved the highest average F1 score of 0.98, followed by Qwen2.5(32B) and DeepSeekR1(32B) with F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistral(7B) and DeepSeekR1(7B) being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports. Our findings demonstrate the feasibility of implementing open-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fast and resource-efficient diagnostic categorization.
Related
- Extraction of clinical information from faxed medical records using a small local large language model pipeline on consumer hardware
- Comparative Analysis of Traditional and Modern Extraction Techniques for Phenolic Compounds From Leontopodium and Optimization of Extraction Process Using Response Surface Methodology.
- Treating diagnostic frameworks as information systems: normative implications and opportunities
- Comparative outcomes of natural orifice specimen extraction surgery versus conventional minilaparotomy-assisted specimen extraction in colorectal cancer: A single-center retrospective study.
- Comparative evaluation of two automated magnetic-bead cfDNA extraction workflows in early colorectal cancer
BibTeX
@article{amirrajab2026comparative,
title = {{Comparative analysis of open-source locally deployed LLMs for extraction of diagnostic information from clinical CMR imaging reports}},
author = {Sina Amirrajab and Volker Vehof and Michael Bietenbeck and Ali Yilmaz},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-75150-7},
url = {https://www.nature.com/articles/s41598-026-75150-7},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Comparative analysis of open-source locally deployed LLMs for extraction of diagnostic information from clinical CMR imaging reports
AU - Sina Amirrajab
AU - Volker Vehof
AU - Michael Bietenbeck
AU - Ali Yilmaz
PY - 2026
DA - 2026/10/14
JO - Scientific Reports
DO - 10.1038/s41598-026-75150-7
UR - https://www.nature.com/articles/s41598-026-75150-7
AB - We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. We evaluated nine open-source LLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models. Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories. Google’s Gemma 2(27B) model achieved the highest average F1 score of 0.98, followed by Qwen2.5(32B) and DeepSeekR1(32B) with F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistral(7B) and DeepSeekR1(7B) being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports. Our findings demonstrate the feasibility of implementing open-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fast and resource-efficient diagnostic categorization.
SN - 2045-2322
ER -