Visual speech enhances phoneme separability in human superior temporal gyrus
- 1University of Michigan, Ann Arbor
Abstract
Visual speech, such as lipreading, facilitates spoken word recognition, but the neural mechanisms underlying audiovisual speech perception remain poorly understood. Visual cues may disambiguate fine-grained articulatory features during early perceptual stages or instead integrate with speech at more categorical, phoneme-level stages. To test how speech representations are modulated by visual input, we analyzed intracranial electroencephalography (iEEG) signals recorded from 12 epilepsy patients performing an audiovisual speech perception task. Participants perceived 16 monosyllabic words presented in auditory-only, visual-only, or congruent audiovisual formats. Words were constructed from four onset consonants (/b/, /g/, /m/, /n/) and four rimes (vowel nucleus and any coda consonants). We examined event-related potentials (ERP) in superior temporal gyrus (STG) and trained support vector machine (SVM) classifiers to decode word identity from neural activity at individual electrodes. Discrete and continuous confusion matrices captured complementary changes in classification accuracy and normalized inverse classification loss, a continuous proxy for classifier confidence. Decoding performance was hierarchically evaluated at the word, phoneme, and phonetic feature levels to determine the representations affected by visual speech. Congruent audiovisual speech increased classifier confidence for phoneme-level representations and improved decoding accuracy at both the word and phoneme level, without corresponding effects on phonetic features. Time-resolved analyses further revealed earlier successful decoding for audiovisual than auditory-only speech, with audiovisual enhancement primarily observed for onset consonants rather than rimes. Together, these findings suggest that visual speech sharpens primarily categorical phoneme representations in STG, with accelerated speech processing and improved word recognition emerging as downstream consequences of phoneme-level enhancement.
Related
- The (Butter)fly Dream: Linear Separability of World-Model Rollouts in a Connectome-Constrained Visual Model
- Distinct processing of visual motion by transcriptomic cell types in the superior colliculus
- Top-down semantic predictions align phonetic neuronal dynamics in human superior temporal gyrus
- The superior temporal asymmetrical pit remains asymmetric in autism spectrum disorder
- Attention selectively enhances collision sensitivity in the superficial SC, PBGN and centromedial amygdala to looming stimuli in the upper visual field
BibTeX
@misc{li2026visual,
title = {{Visual speech enhances phoneme separability in human superior temporal gyrus}},
author = {Y. Li and I. DeWitt and J. R. Brennan and V. S. Wasade and W. Stacey and D. Brang},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.10.750762},
url = {https://www.biorxiv.org/content/10.64898/2026.09.10.750762v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Visual speech enhances phoneme separability in human superior temporal gyrus
AU - Y. Li
AU - I. DeWitt
AU - J. R. Brennan
AU - V. S. Wasade
AU - W. Stacey
AU - D. Brang
PY - 2026
DA - 2026/09/10
PB - bioRxiv
DO - 10.64898/2026.09.10.750762
UR - https://www.biorxiv.org/content/10.64898/2026.09.10.750762v1
AB - Visual speech, such as lipreading, facilitates spoken word recognition, but the neural mechanisms underlying audiovisual speech perception remain poorly understood. Visual cues may disambiguate fine-grained articulatory features during early perceptual stages or instead integrate with speech at more categorical, phoneme-level stages. To test how speech representations are modulated by visual input, we analyzed intracranial electroencephalography (iEEG) signals recorded from 12 epilepsy patients performing an audiovisual speech perception task. Participants perceived 16 monosyllabic words presented in auditory-only, visual-only, or congruent audiovisual formats. Words were constructed from four onset consonants (/b/, /g/, /m/, /n/) and four rimes (vowel nucleus and any coda consonants). We examined event-related potentials (ERP) in superior temporal gyrus (STG) and trained support vector machine (SVM) classifiers to decode word identity from neural activity at individual electrodes. Discrete and continuous confusion matrices captured complementary changes in classification accuracy and normalized inverse classification loss, a continuous proxy for classifier confidence. Decoding performance was hierarchically evaluated at the word, phoneme, and phonetic feature levels to determine the representations affected by visual speech. Congruent audiovisual speech increased classifier confidence for phoneme-level representations and improved decoding accuracy at both the word and phoneme level, without corresponding effects on phonetic features. Time-resolved analyses further revealed earlier successful decoding for audiovisual than auditory-only speech, with audiovisual enhancement primarily observed for onset consonants rather than rimes. Together, these findings suggest that visual speech sharpens primarily categorical phoneme representations in STG, with accelerated speech processing and improved word recognition emerging as downstream consequences of phoneme-level enhancement.
ER -