Fluent but not fail-safe: machine-translated radiology reports still need expert review.
- 1Clinic for Diagnostic and Interventional Radiology, University Hospital Heidelberg, Heidelberg, Germany.
- 2Clinic for Diagnostic and Interventional Radiology, University Hospital Heidelberg, Heidelberg, Germany. [email protected].
Abstract
Objectives
Cross-border care, teleradiology, and international referral pathways increasingly require radiology report translation, but isolated mistranslations may alter clinical meaning. We evaluated German-to-English machine translation of radiology reports and correction-based localization of text requiring review.Materials and methods
In this retrospective study, 310 de-identified German radiology reports from June and July 2025 were translated into English with DeepL. Eight radiologists assigned 5-point quality scores and corrected perceived errors. Thirty reports were reviewed by all eight readers, and 280 reports were assigned for single-reader evaluation, yielding 520 evaluations. Automated metrics and large language model judges were tested for report-level identification of translations requiring review; a correction-based segment-level edit-prediction model was evaluated on held-out shared reports.Results
Most translations were rated good or excellent (339/520, 65.2%), but clinically important mistranslations involved diagnosis, anatomy, and radiologic terminology. Agreement was low for ordinal quality scores (Krippendorff α = 0.078, p = 0.006) but high for expert corrections (pairwise "bilingual evaluation understudy"-BLEU = 83.0, p < 0.001). In the conservative primary screen of 30 shared reports, no automated method discriminated translations requiring review (area under receiver operating characteristic curve [AUC] = 0.51-0.65; p = 0.100-0.920). The final edit-prediction model achieved a concordance index of 0.697 and an AUC of 0.731 for segments edited by at least six of eight readers.Conclusion
Machine-translated radiology reports were often acceptable on average, but translations containing errors requiring expert review persisted. Expert corrections were more reproducible than ordinal quality scores and supported localization of text requiring human review.Key points
Question Cross-border radiology workflows increasingly require report translation, but fluent machine translations may still contain meaning-changing errors requiring expert review for safe care. Findings Most machine translations were rated good or excellent, yet clinically important mistranslations occurred, and automated report-level metrics did not reliably identify reports needing review. Relevance statement Machine translation may support cross-border radiology workflows, but clinically important mistranslations in otherwise fluent reports require human-in-the-loop oversight and tools that localize text requiring expert review.
Related
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Multi-Scale MRI, Radiology Reports, and Blood Biomarker–Guided Multimodal Deep Learning for Predicting Postoperative Recurrence in Cervical Cancer: A Multicenter Study
- Accuracy of Provisional Reports by Radiology Residents on Call.
- Psychological factors contribute more to chronic low back pain than spine pathology: An LLM-based analysis of radiology reports
- Natural Language Processing Based Solution for Labeling Brain Metastasis Identified in Radiology Reports
BibTeX
@article{seibold2026fluent,
title = {{Fluent but not fail-safe: machine-translated radiology reports still need expert review.}},
author = {Constantin Seibold and Timo Leichenich and Patrick Stein and Edem Atsiatorme and Patrick Kliesch and Gökçe A Sevim and Peter Trauth and Alexander Kremer and Arved Bischoff and Hans-Ulrich Kauczor and Matthias A Fink},
journal = {European radiology experimental},
year = {2026},
month = oct,
doi = {10.1186/s41747-026-00827-7},
url = {https://europepmc.org/article/MED/42873404},
note = {CC BY},
}
RIS
TY - JOUR
TI - Fluent but not fail-safe: machine-translated radiology reports still need expert review.
AU - Constantin Seibold
AU - Timo Leichenich
AU - Patrick Stein
AU - Edem Atsiatorme
AU - Patrick Kliesch
AU - Gökçe A Sevim
AU - Peter Trauth
AU - Alexander Kremer
AU - Arved Bischoff
AU - Hans-Ulrich Kauczor
AU - Matthias A Fink
PY - 2026
DA - 2026/10/14
JO - European radiology experimental
DO - 10.1186/s41747-026-00827-7
UR - https://europepmc.org/article/MED/42873404
AB - Objectives Cross-border care, teleradiology, and international referral pathways increasingly require radiology report translation, but isolated mistranslations may alter clinical meaning. We evaluated German-to-English machine translation of radiology reports and correction-based localization of text requiring review.Materials and methods In this retrospective study, 310 de-identified German radiology reports from June and July 2025 were translated into English with DeepL. Eight radiologists assigned 5-point quality scores and corrected perceived errors. Thirty reports were reviewed by all eight readers, and 280 reports were assigned for single-reader evaluation, yielding 520 evaluations. Automated metrics and large language model judges were tested for report-level identification of translations requiring review; a correction-based segment-level edit-prediction model was evaluated on held-out shared reports.Results Most translations were rated good or excellent (339/520, 65.2%), but clinically important mistranslations involved diagnosis, anatomy, and radiologic terminology. Agreement was low for ordinal quality scores (Krippendorff α = 0.078, p = 0.006) but high for expert corrections (pairwise "bilingual evaluation understudy"-BLEU = 83.0, p < 0.001). In the conservative primary screen of 30 shared reports, no automated method discriminated translations requiring review (area under receiver operating characteristic curve [AUC] = 0.51-0.65; p = 0.100-0.920). The final edit-prediction model achieved a concordance index of 0.697 and an AUC of 0.731 for segments edited by at least six of eight readers.Conclusion Machine-translated radiology reports were often acceptable on average, but translations containing errors requiring expert review persisted. Expert corrections were more reproducible than ordinal quality scores and supported localization of text requiring human review.Key points Question Cross-border radiology workflows increasingly require report translation, but fluent machine translations may still contain meaning-changing errors requiring expert review for safe care. Findings Most machine translations were rated good or excellent, yet clinically important mistranslations occurred, and automated report-level metrics did not reliably identify reports needing review. Relevance statement Machine translation may support cross-border radiology workflows, but clinically important mistranslations in otherwise fluent reports require human-in-the-loop oversight and tools that localize text requiring expert review.
SN - 2509-9280
ER -