Large Language Model-Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation.
7 affiliations
- 1Department of Radiology, The Second Affiliated Hospital of Kunming Medical University, Kunming, Yunnan, 650101, China, 86 0871-63402918.
- 2Department of Radiology, The Sixth Affiliated Hospital of Kunming Medical University, Yuxi, Yunnan, China.
- 3School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China.
- 4School of Biomedical Engineering, ShanghaiTech University, Shanghai, China.
- 5Department of Radiology, School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom.
- 6Department of Radiology, National Center for Cardiovascular Diseases, Fuwai Hospital, Beijing, China.
- 7Department of Radiology, Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, United States.
Abstract
Background
Radiology reports serve as the definitive clinical record of imaging diagnoses, with their accuracy being directly critical to patient safety. However, amid escalating workloads, manual review remains inadequate for capturing subtle errors in reports on a large scale. Although large language models (LLMs) have demonstrated considerable promise in medical text processing, previous studies have predominantly been confined to radiology report text from a single imaging modality or datasets with synthetic errors.Objective
This study aimed to evaluate the feasibility, accuracy, and generalizability of general-purpose LLMs for quality assurance of radiology report text from multiple imaging modalities (x-ray, computed tomography [CT], and magnetic resonance imaging [MRI]), multi-anatomical regions, and multicenter clinical settings, with head-to-head comparisons conducted against radiologists of varying seniority.Methods
Radiology report texts from 3 aforementioned imaging modalities and 5 anatomical regions (head, chest, abdomen, spine, and knee) were retrospectively collected from center 1 (March 2025-May 2025) and reviewed using a clinician-annotated 5-category error taxonomy (type 1: interpretive; type 2: factual). Five LLMs (Qwen3, GPT-5.4 Pro, Gemini 3.1 Pro, Llama 4, and DeepSeek-V3.1) were benchmarked on 10,000 randomly selected reports, and the best-performing model was evaluated on the full cohort. A blinded head-to-head comparison with 3 radiologists (3-10 years of experience) used a modality-stratified, error-enriched challenge set. The radiologists subsequently rated the model-generated explanations using a 5-point Likert scale. Cross-center performance was evaluated in an independent external cohort. The same 2 senior radiologists labeled reports from both centers using the same protocol, resolving disagreements by consensus.Results
In total, 4.6% (4431/95,582) of the reports in center 1 contained at least one confirmed text internal error. Qwen3, the selected model, achieved a report-level sensitivity, specificity, positive predictive value, and F₁-score of 0.99, 0.99, 0.90, and 0.94, respectively, with high performance across reports from the 3 imaging modalities and multiple anatomical regions and better detection of factual than interpretive errors. In the error-enriched challenge set, its error instance exact category recall exceeded that of the radiologists (0.96 vs 0.45-0.67; P<.001), with a shorter mean task completion time (5.80 vs 55.80-67.80 seconds per report). Explanations had a mean Likert score of 4.57, and all inter- and intraobserver intraclass correlation coefficients exceeded 0.89. In center 2, balanced accuracy was 0.94 (95% CI 0.92-0.95). Before consensus, Cohen κ for report-level error presence was 0.783 (center 1) and 0.850 (center 2).Conclusions
This study suggests that LLMs can serve as efficient adjunctive tools for text-level quality control of radiology reports, particularly for detecting explicit factual errors. Performance remained stable across centers. In the text-only review task, the model achieved higher exact category recall and shorter task completion times than the participating radiologists. These findings support the potential value of LLM-assisted radiology report text review in real-world clinical environments.
Related
- Multi-Scale MRI, Radiology Reports, and Blood Biomarker–Guided Multimodal Deep Learning for Predicting Postoperative Recurrence in Cervical Cancer: A Multicenter Study
- Sycophancy in Large Language Models in Oral and Maxillofacial Radiology: The Impact of Misleading Context on Image- and Text-Based Diagnostic Scenarios
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Large Language Models in the Radiology Workflow: Current Applications, Challenges, and Future Perspectives.
- Factors associated with choosing diagnostic radiology as a future specialty among Palestinian medical students: a multicenter cross-sectional study
BibTeX
@article{wang2026large,
title = {{Large Language Model-Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation.}},
author = {Lujing Wang and Yushuang Zhang and Liang Peng and Pei Liu and Li Wang and Yu Cui and Yixin Chen and Xingyu Li and Yitian Tao and Xiuxian Gong and Xiaoying Zhao and Yujiao Song and Jian Chen and Shihua Zhao and Fuying Wang and Jian Ma and Xinxiang Zhao},
journal = {Journal of medical Internet research},
year = {2026},
month = oct,
doi = {10.2196/106250},
url = {https://europepmc.org/article/MED/42954415},
note = {CC BY},
}
RIS
TY - JOUR
TI - Large Language Model-Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation.
AU - Lujing Wang
AU - Yushuang Zhang
AU - Liang Peng
AU - Pei Liu
AU - Li Wang
AU - Yu Cui
AU - Yixin Chen
AU - Xingyu Li
AU - Yitian Tao
AU - Xiuxian Gong
AU - Xiaoying Zhao
AU - Yujiao Song
AU - Jian Chen
AU - Shihua Zhao
AU - Fuying Wang
AU - Jian Ma
AU - Xinxiang Zhao
PY - 2026
DA - 2026/10/28
JO - Journal of medical Internet research
DO - 10.2196/106250
UR - https://europepmc.org/article/MED/42954415
AB - Background Radiology reports serve as the definitive clinical record of imaging diagnoses, with their accuracy being directly critical to patient safety. However, amid escalating workloads, manual review remains inadequate for capturing subtle errors in reports on a large scale. Although large language models (LLMs) have demonstrated considerable promise in medical text processing, previous studies have predominantly been confined to radiology report text from a single imaging modality or datasets with synthetic errors.Objective This study aimed to evaluate the feasibility, accuracy, and generalizability of general-purpose LLMs for quality assurance of radiology report text from multiple imaging modalities (x-ray, computed tomography [CT], and magnetic resonance imaging [MRI]), multi-anatomical regions, and multicenter clinical settings, with head-to-head comparisons conducted against radiologists of varying seniority.Methods Radiology report texts from 3 aforementioned imaging modalities and 5 anatomical regions (head, chest, abdomen, spine, and knee) were retrospectively collected from center 1 (March 2025-May 2025) and reviewed using a clinician-annotated 5-category error taxonomy (type 1: interpretive; type 2: factual). Five LLMs (Qwen3, GPT-5.4 Pro, Gemini 3.1 Pro, Llama 4, and DeepSeek-V3.1) were benchmarked on 10,000 randomly selected reports, and the best-performing model was evaluated on the full cohort. A blinded head-to-head comparison with 3 radiologists (3-10 years of experience) used a modality-stratified, error-enriched challenge set. The radiologists subsequently rated the model-generated explanations using a 5-point Likert scale. Cross-center performance was evaluated in an independent external cohort. The same 2 senior radiologists labeled reports from both centers using the same protocol, resolving disagreements by consensus.Results In total, 4.6% (4431/95,582) of the reports in center 1 contained at least one confirmed text internal error. Qwen3, the selected model, achieved a report-level sensitivity, specificity, positive predictive value, and F₁-score of 0.99, 0.99, 0.90, and 0.94, respectively, with high performance across reports from the 3 imaging modalities and multiple anatomical regions and better detection of factual than interpretive errors. In the error-enriched challenge set, its error instance exact category recall exceeded that of the radiologists (0.96 vs 0.45-0.67; P<.001), with a shorter mean task completion time (5.80 vs 55.80-67.80 seconds per report). Explanations had a mean Likert score of 4.57, and all inter- and intraobserver intraclass correlation coefficients exceeded 0.89. In center 2, balanced accuracy was 0.94 (95% CI 0.92-0.95). Before consensus, Cohen κ for report-level error presence was 0.783 (center 1) and 0.850 (center 2).Conclusions This study suggests that LLMs can serve as efficient adjunctive tools for text-level quality control of radiology reports, particularly for detecting explicit factual errors. Performance remained stable across centers. In the text-only review task, the model achieved higher exact category recall and shorter task completion times than the participating radiologists. These findings support the potential value of LLM-assisted radiology report text review in real-world clinical environments.
SN - 1439-4456
SN - 1438-8871
ER -