Human-calibrated automated intelligibility evaluation of speech neuroprostheses
Abstract
Objective: Brain-to-voice neuroprostheses can synthesize speech from the neural activity of people who have lost the ability to speak, enabling more natural and expressive communication. Assessing the intelligibility of synthesized speech, however, remains challenging because there is no ground-truth acoustic signal. The field instead relies on human transcription, which is costly and difficult to scale or compare across studies. Automatic speech recognition (ASR) offers a potential alternative, but state-of-the-art ASR models are optimized for transcription accuracy on typical fluent speech and may not reflect how humans perceive atypical synthesized speech. Approach: We collected human transcriptions for ~14.5 hours of synthesized speech from intracortical brain-to-voice decoders for one participant (yielding ~72 hours of transcribed speech). We used the human-transcribed synthesized speech to fine-tune Whisper-large-v3 with low-rank adaptation (LoRA) so that its word error rate (WER) approximates human WER. We then evaluated the model on held-out sentences. Main results: Fine-tuning increased the Pearson correlation between model WER and human WER from 0.62 to 0.76. The fine-tuned model's WER closely tracked human WER across evaluation sets, including speech synthesized by unseen brain-to-voice decoders. Close agreement with human WER was achieved with only 1.5 hours of synthesized speech. Significance: These results demonstrate that human-calibrated ASR can provide a scalable measure of brain-to-voice intelligibility that reflects human transcription more closely than zero-shot ASR does.
Related
- Speech Intelligibility Index (SII) as a Potential Referral Metric for Adult Cochlear Implant Candidacy Evaluation
- Modeling Paralinguistic Features for Speech Neuroprostheses: A Systematic Review
- Rclade: automated taxonomic collapsing and geological-timescale annotation of time-calibrated phylogenetic trees in R
- Analytical validation of digital measure of intelligibility in progressive supranuclear palsy
- Development and Evaluation of a Realistic Audio-Visual Multi-Talker Speech Comprehension Test
BibTeX
@misc{srinivasan2026humancalibrated,
title = {{Human-calibrated automated intelligibility evaluation of speech neuroprostheses}},
author = {Aparna Srinivasan and Maitreyee Wairagkar and Ramie Wakeb and Dana Farias and Lee M. Miller and David M. Brandman and Sergey D. Stavisky},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.27.762196},
url = {https://www.biorxiv.org/content/10.64898/2026.10.27.762196v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Human-calibrated automated intelligibility evaluation of speech neuroprostheses
AU - Aparna Srinivasan
AU - Maitreyee Wairagkar
AU - Ramie Wakeb
AU - Dana Farias
AU - Lee M. Miller
AU - David M. Brandman
AU - Sergey D. Stavisky
PY - 2026
DA - 2026/10/27
PB - bioRxiv
DO - 10.64898/2026.10.27.762196
UR - https://www.biorxiv.org/content/10.64898/2026.10.27.762196v1
AB - Objective: Brain-to-voice neuroprostheses can synthesize speech from the neural activity of people who have lost the ability to speak, enabling more natural and expressive communication. Assessing the intelligibility of synthesized speech, however, remains challenging because there is no ground-truth acoustic signal. The field instead relies on human transcription, which is costly and difficult to scale or compare across studies. Automatic speech recognition (ASR) offers a potential alternative, but state-of-the-art ASR models are optimized for transcription accuracy on typical fluent speech and may not reflect how humans perceive atypical synthesized speech. Approach: We collected human transcriptions for ~14.5 hours of synthesized speech from intracortical brain-to-voice decoders for one participant (yielding ~72 hours of transcribed speech). We used the human-transcribed synthesized speech to fine-tune Whisper-large-v3 with low-rank adaptation (LoRA) so that its word error rate (WER) approximates human WER. We then evaluated the model on held-out sentences. Main results: Fine-tuning increased the Pearson correlation between model WER and human WER from 0.62 to 0.76. The fine-tuned model's WER closely tracked human WER across evaluation sets, including speech synthesized by unseen brain-to-voice decoders. Close agreement with human WER was achieved with only 1.5 hours of synthesized speech. Significance: These results demonstrate that human-calibrated ASR can provide a scalable measure of brain-to-voice intelligibility that reflects human transcription more closely than zero-shot ASR does.
SN - 2692-8205
ER -