Large language model assistance for report-based carotid plaque AHA classification: a multi-reader study
Abstract
Objectives: To evaluate whether large language model assistance improves radiologists’ agreement with a text-based reference standard in inferring the modified American Heart Association carotid plaque classification from free-text MRI reports, and whether a lightweight model can perform well under criteria-guided prompting.
Materials and methods: This retrospective study included 433 patients from three institutions. In a report-based reader study, four radiologists classified MRI reports with and without large language model assistance. In standalone evaluation, a rule-based baseline and 10 models were compared under naive and criteria-guided prompting. The reference standard was expert consensus based on the report text.
Results: Among 433 patients (mean age, 65.9 years ± 13.4; 248 males), AI assistance improved six-class agreement with the text-based reference standard. At the vessel level, junior radiologists improved from 61.75% to 83.12% (Qwen3-8B) and 87.75% (DeepSeek-R1); senior radiologists from 77.00% to 87.50% and 92.25%, respectively (all Holm-corrected p < 0.001). Patient-level agreement increased from 41.0% to 78.8% (junior) and from 59.8% to 87.8% (senior) with DeepSeek-R1. Agreement increased (quadratic-weighted κ : 0.231 without AI vs 0.775 with DeepSeek-R1). In standalone evaluation, criteria-guided prompting improved Qwen3-8B agreement from 38.91% to 81.52% (95% CI: 78.87–84.06%), remaining 5.89 percentage points below DeepSeek-R1 (87.41%; 95% CI: 85.10–89.61%).
Conclusion: In this report-based classification setting, large language model assistance improved radiologists’ agreement with the text-based reference standard, with larger gains in junior radiologists. When prompts explicitly encoded AHA criteria, an 8B local candidate model performed well, supporting its potential use as a standardization aid for text-based workflows.
Key Points: Question Report-based modified AHA classification of carotid plaques is experience-dependent and difficult to standardize, as mapping multi-sequence imaging findings to classification conclusions imposes substantial cognitive load.
Findings Large language model assistance improved agreement with a text-based reference standard and raised inter-reader consistency (weighted κ 0.231–0.775), narrowing the junior–senior gap.
Critical relevance statement Embedding explicit AHA criteria in the prompt let an 8B model approach 671B-model agreement (38.91–81.52%), supporting standardized text-based workflows without fine-tuning.
Related
- Transient perivascular inflammation of the carotid artery superimposed on carotid atherosclerotic plaque, recurring at the mid-common carotid artery: a case report
- The initial application of RawVisionTM plaque (RVP) quantitative technique in evaluation for carotid plaque vulnerability
- Carotid atherosclerotic plaque characterization using dual-energy computed tomography angiography and its correlation with cerebral ischemic stroke
- Quantitative CTA-Based Carotid Plaque Composition Across Stroke Etiologies and Residual Vascular Risk
- Predictive value of plasma soluble thrombomodulin for new-onset carotid plaque in adults with carotid intima-media thickening.
BibTeX
@article{wang2026large,
title = {{Large language model assistance for report-based carotid plaque AHA classification: a multi-reader study}},
author = {Mengfei Wang and Yixiao Tang and Yulu Yang and Yifan Tu and Zhongzheng Cao and Runjianya Ling and Kaiyue Sun and Jian Ma and Zhicheng Zhang and Ruipeng Zhang and Xiaoer Wei and Dan Wang and Yuehua Li},
journal = {Insights into imaging},
year = {2026},
month = oct,
doi = {10.1186/s13244-026-02421-2},
url = {https://europepmc.org/article/PMC/PMC13675435},
note = {CC BY},
}
RIS
TY - JOUR
TI - Large language model assistance for report-based carotid plaque AHA classification: a multi-reader study
AU - Mengfei Wang
AU - Yixiao Tang
AU - Yulu Yang
AU - Yifan Tu
AU - Zhongzheng Cao
AU - Runjianya Ling
AU - Kaiyue Sun
AU - Jian Ma
AU - Zhicheng Zhang
AU - Ruipeng Zhang
AU - Xiaoer Wei
AU - Dan Wang
AU - Yuehua Li
PY - 2026
DA - 2026/10/16
JO - Insights into imaging
DO - 10.1186/s13244-026-02421-2
UR - https://europepmc.org/article/PMC/PMC13675435
AB - Objectives: To evaluate whether large language model assistance improves radiologists’ agreement with a text-based reference standard in inferring the modified American Heart Association carotid plaque classification from free-text MRI reports, and whether a lightweight model can perform well under criteria-guided prompting. Materials and methods: This retrospective study included 433 patients from three institutions. In a report-based reader study, four radiologists classified MRI reports with and without large language model assistance. In standalone evaluation, a rule-based baseline and 10 models were compared under naive and criteria-guided prompting. The reference standard was expert consensus based on the report text. Results: Among 433 patients (mean age, 65.9 years ± 13.4; 248 males), AI assistance improved six-class agreement with the text-based reference standard. At the vessel level, junior radiologists improved from 61.75% to 83.12% (Qwen3-8B) and 87.75% (DeepSeek-R1); senior radiologists from 77.00% to 87.50% and 92.25%, respectively (all Holm-corrected p < 0.001). Patient-level agreement increased from 41.0% to 78.8% (junior) and from 59.8% to 87.8% (senior) with DeepSeek-R1. Agreement increased (quadratic-weighted κ : 0.231 without AI vs 0.775 with DeepSeek-R1). In standalone evaluation, criteria-guided prompting improved Qwen3-8B agreement from 38.91% to 81.52% (95% CI: 78.87–84.06%), remaining 5.89 percentage points below DeepSeek-R1 (87.41%; 95% CI: 85.10–89.61%). Conclusion: In this report-based classification setting, large language model assistance improved radiologists’ agreement with the text-based reference standard, with larger gains in junior radiologists. When prompts explicitly encoded AHA criteria, an 8B local candidate model performed well, supporting its potential use as a standardization aid for text-based workflows. Key Points: Question Report-based modified AHA classification of carotid plaques is experience-dependent and difficult to standardize, as mapping multi-sequence imaging findings to classification conclusions imposes substantial cognitive load. Findings Large language model assistance improved agreement with a text-based reference standard and raised inter-reader consistency (weighted κ 0.231–0.775), narrowing the junior–senior gap. Critical relevance statement Embedding explicit AHA criteria in the prompt let an 8B model approach 671B-model agreement (38.91–81.52%), supporting standardized text-based workflows without fine-tuning.
SN - 1869-4101
ER -