Beyond word error rate: clinical risk as the necessary standard for ambient AI scribe evaluation: evidence from 77 global languages
Abstract
ObjectiveAmbient AI scribes evaluated using frequency-based metrics such as word error rate (WER), which do not represent clinical consequence. We tested whether variation in these metrics tracks consequential transcription errors.
MethodsWe constructed a multilingual corpus from five clinical dictation scripts spanning a complexity gradient, translated into 99 languages, rendered to synthetic speech under three acoustic conditions, and transcribed by a production ambient scribe. Six frequency metrics were computed. Three independent large language model raters from external providers assessed clinically meaningful error patterns in context using a Severity x Likelihood framework informed by UK digital clinical-safety-risk-management principles.
ResultsAcross 59,819 genuine transcription-error occurrences, 58,329 (97.5%) were LOW risk and 251 (0.42%) CRITICAL or HIGH. None of six frequency metrics showed a statistically detectable association with serious clinical risk across languages; correlations were small (absolute Spearman {rho}<0.16). A Severity x Likelihood sum remained strongly correlated with WER ({rho}=0.80), showing that the aggregate remained dominated by benign errors. At complexity level 3, low-resource languages had worse WER than high-resource languages ({beta}=+0.078, 95% CI +0.045 to +0.111; p<0.0001), without a detectable difference in CRITICAL/HIGH risk (OR 1.21, 95% CI 0.43 to 3.43; p=0.72).
Consultation complexity was the principal predictor of serious risk (OR 3.06 per level, p<0.0001).
ConclusionAcross this controlled multilingual corpus, aggregate transcription-frequency metrics did not reliably track the sparse severe tail of clinically consequential errors. WER remains appropriate for transcription quality, but these data do not support its use alone as a proxy for clinical safety.
Context-aware assessment of error consequence provides complementary information that frequency measures can dilute.
HighlightsO_LIWER did not track serious clinical-risks rate across 77 languages C_LIO_LIFive related frequency metrics showed the same dissociation C_LIO_LISeverity-weighting remained dominated by common low-risk errors C_LIO_LIQuality tracked language resource; serious risk tracked complexity C_LIO_LIWe provide a reusable context-aware clinical-risk instrument C_LI
Related
- Thrombophilia Testing After VTE and in Related Clinical Contexts: Is It Necessary, and Does It Change Management? Appraisal of Evidence, Guidelines, and Clinical Yield
- Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support
- When should bladder cancer prompt evaluation for Lynch syndrome? An evidence-informed clinical framework
- An Unfinished but Necessary Path: Ring-Based Cuffless Blood Pressure Monitoring at the Crossroads of Innovation and Clinical Reality
- Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non‐English Languages
BibTeX
@misc{bergman2026beyond,
title = {{Beyond word error rate: clinical risk as the necessary standard for ambient AI scribe evaluation: evidence from 77 global languages}},
author = {H. Bergman and V. Liu and B. Austin and R. Sanghera},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.10.26362715},
url = {https://www.medrxiv.org/content/10.64898/2026.09.10.26362715v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Beyond word error rate: clinical risk as the necessary standard for ambient AI scribe evaluation: evidence from 77 global languages
AU - H. Bergman
AU - V. Liu
AU - B. Austin
AU - R. Sanghera
PY - 2026
DA - 2026/09/10
PB - medRxiv
DO - 10.64898/2026.09.10.26362715
UR - https://www.medrxiv.org/content/10.64898/2026.09.10.26362715v1
AB - ObjectiveAmbient AI scribes evaluated using frequency-based metrics such as word error rate (WER), which do not represent clinical consequence. We tested whether variation in these metrics tracks consequential transcription errors. MethodsWe constructed a multilingual corpus from five clinical dictation scripts spanning a complexity gradient, translated into 99 languages, rendered to synthetic speech under three acoustic conditions, and transcribed by a production ambient scribe. Six frequency metrics were computed. Three independent large language model raters from external providers assessed clinically meaningful error patterns in context using a Severity x Likelihood framework informed by UK digital clinical-safety-risk-management principles. ResultsAcross 59,819 genuine transcription-error occurrences, 58,329 (97.5%) were LOW risk and 251 (0.42%) CRITICAL or HIGH. None of six frequency metrics showed a statistically detectable association with serious clinical risk across languages; correlations were small (absolute Spearman {rho}<0.16). A Severity x Likelihood sum remained strongly correlated with WER ({rho}=0.80), showing that the aggregate remained dominated by benign errors. At complexity level 3, low-resource languages had worse WER than high-resource languages ({beta}=+0.078, 95% CI +0.045 to +0.111; p<0.0001), without a detectable difference in CRITICAL/HIGH risk (OR 1.21, 95% CI 0.43 to 3.43; p=0.72). Consultation complexity was the principal predictor of serious risk (OR 3.06 per level, p<0.0001). ConclusionAcross this controlled multilingual corpus, aggregate transcription-frequency metrics did not reliably track the sparse severe tail of clinically consequential errors. WER remains appropriate for transcription quality, but these data do not support its use alone as a proxy for clinical safety. Context-aware assessment of error consequence provides complementary information that frequency measures can dilute. HighlightsO_LIWER did not track serious clinical-risks rate across 77 languages C_LIO_LIFive related frequency metrics showed the same dissociation C_LIO_LISeverity-weighting remained dominated by common low-risk errors C_LIO_LIQuality tracked language resource; serious risk tracked complexity C_LIO_LIWe provide a reusable context-aware clinical-risk instrument C_LI
ER -