Benchmarking Speech Recognition Models for Medical Consultations in Latin American Spanish: A Comparative Evaluation with Fine-Tuning
- 1Emory University
Abstract
BACKGROUNDArtificial intelligence (AI) medical scribes rely on speech-to-text (STT) models for transcription. Evaluations of STT models in non-English settings remain scarce. We benchmarked ten STT models on medical consultations from Latin American (LatAm) Spanish and assessed whether fine-tuning improves transcription accuracy.
METHODSTen videos depicting medical consultations. Human transcriptions were the ground truth. Five closed-source models were evaluated: gpt-4o-transcribe, gpt-4o-mini-transcribe, gemini-2.5-pro, Eleven Labs, and Assembly AI; and so were five open-source models: Whisper Large, Whisper Large v3, Whisper Large v3 Turbo, Voxtral Mini 3B, and Canary 1B v2. Whisper Large v3, an open-source model, was fine-tuned. One video was withheld from training. Performance assessed using Word Error Rate (WER), Character Error Rate (CER), BLEU Score, ROUGE-L, BERT Score, and Semantic Similarity on the one withheld video.
RESULTSNone of the fine-tuning iterations outperformed the vanilla Whisper Large v3. When evaluated on the held-out test video, the fine-tuned open-source Whisper Large v3 did not outperform any of the top closed-source models; conversely, relative to other open-source models, the fine-tuned Whisper Large v3 had superior performance across all metrics, such as in BLEU score (63% vs. 58% for the second-best model), BERT Score (89% vs. 86%), and semantic similarity (89% vs. 83%).
CONCLUSIONSWhisper Large v3 and its fine-tuned variant are the best open-source STT models for transcribing medical conversations in LatAm Spanish. These findings provide evidence for developing AI medical scribes tailored to Spanish-speaking LatAm.
Related
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- Delayed Recognition of Autism Spectrum Disorder and Off-Label Lurasidone After Intolerance to Risperidone and Aripiprazole in a Latin American Child: A Case Report
- Evaluation of an Individual Learning Opportunity Utilizing an Asian American Studies-Themed Mental Health Workbook
- Benchmarking Open-Source Vision–Language Models for Food Image Recognition and Portion Size Estimation
- Effect of In-line Clinical Decision Support to Improve Accuracy of American Society of Anesthesiologists Physical Status Documentation: A Two Center Evaluation of a Quality Intervention
BibTeX
@misc{carrillolarco2026benchmarking,
title = {{Benchmarking Speech Recognition Models for Medical Consultations in Latin American Spanish: A Comparative Evaluation with Fine-Tuning}},
author = {R. M. Carrillo-Larco and A. Carbajal Serrano and P. S. Condori Pinedo},
howpublished = {medRxiv},
year = {2026},
month = jul,
doi = {10.64898/2026.07.14.26358062},
url = {https://www.medrxiv.org/content/10.64898/2026.07.14.26358062v3},
note = {CC BY},
}
RIS
TY - UNPB
TI - Benchmarking Speech Recognition Models for Medical Consultations in Latin American Spanish: A Comparative Evaluation with Fine-Tuning
AU - R. M. Carrillo-Larco
AU - A. Carbajal Serrano
AU - P. S. Condori Pinedo
PY - 2026
DA - 2026/07/14
PB - medRxiv
DO - 10.64898/2026.07.14.26358062
UR - https://www.medrxiv.org/content/10.64898/2026.07.14.26358062v3
AB - BACKGROUNDArtificial intelligence (AI) medical scribes rely on speech-to-text (STT) models for transcription. Evaluations of STT models in non-English settings remain scarce. We benchmarked ten STT models on medical consultations from Latin American (LatAm) Spanish and assessed whether fine-tuning improves transcription accuracy. METHODSTen videos depicting medical consultations. Human transcriptions were the ground truth. Five closed-source models were evaluated: gpt-4o-transcribe, gpt-4o-mini-transcribe, gemini-2.5-pro, Eleven Labs, and Assembly AI; and so were five open-source models: Whisper Large, Whisper Large v3, Whisper Large v3 Turbo, Voxtral Mini 3B, and Canary 1B v2. Whisper Large v3, an open-source model, was fine-tuned. One video was withheld from training. Performance assessed using Word Error Rate (WER), Character Error Rate (CER), BLEU Score, ROUGE-L, BERT Score, and Semantic Similarity on the one withheld video. RESULTSNone of the fine-tuning iterations outperformed the vanilla Whisper Large v3. When evaluated on the held-out test video, the fine-tuned open-source Whisper Large v3 did not outperform any of the top closed-source models; conversely, relative to other open-source models, the fine-tuned Whisper Large v3 had superior performance across all metrics, such as in BLEU score (63% vs. 58% for the second-best model), BERT Score (89% vs. 86%), and semantic similarity (89% vs. 83%). CONCLUSIONSWhisper Large v3 and its fine-tuned variant are the best open-source STT models for transcribing medical conversations in LatAm Spanish. These findings provide evidence for developing AI medical scribes tailored to Spanish-speaking LatAm.
ER -