Naturalistic Movie Viewing Reveals a Distributed Network Supporting Audiovisual Speech Processing
- 1Northeastern University
Abstract
Traditional paradigms in cognitive neuroscience often fail to capture the richness of real-world experiences. To bridge this gap, researchers increasingly use naturalistic stimuli such as movies, which provide complex, multisensory input. Speech perception is a prime example of such complexity: it is inherently multisensory, involving the integration of auditory cues, lip movements, and facial expressions. Prior studies suggest that the left posterior superior temporal sulcus (pSTS) and superior temporal gyrus (STG) play a critical role in this integration. However, most studies of audiovisual speech have relied on simplified or artificial stimuli, limiting ecological validity. To address these gaps, we analyzed fMRI data from human participants watching two full-length films (500 Days of Summer, Citizenfour) to assess the consistency of multisensory speech processes across distinct genres and narratives. We manually annotated word onset times and modality (audiovisual vs. auditory-only). Across two movies and 38 participants we observed consistent speech-related activation in the pSTS and STG across speech types. Compared to auditory-only speech, audiovisual speech recruited a broad left-lateralized network including the angular gyrus, posterior temporal cortex, temporal pole, and inferior frontal gyrus, along with the right cerebellum. These findings were also consistently present in single-subject data. Importantly, most of the regions observed in our analyses have been implicated in multisensory function in the nonhuman primate literature despite receiving relatively little focus in human neuroimaging. Together our results demonstrate that speech processing during naturalistic viewing recruits a distributed network extending substantially beyond regions traditionally emphasized in audiovisual speech research.
Related
- From Exposure to Experience: Measuring Gaze-Based Visual Experience with Faces During Naturalistic Viewing
- AVMoments-EEG: A Large-Scale EEG Dataset of Naturalistic Audiovisual Event Perception
- Individual differences in eye movements dynamics of free viewing conscious and unconscious naturalistic scenes
- rpSTS Stimulation Disrupts Shared Neural Responses to Social Content during Naturalistic Viewing
- Neural tracking of surprisal and semantic distance in naturalistic movie viewing
BibTeX
@misc{smith2026naturalistic,
title = {{Naturalistic Movie Viewing Reveals a Distributed Network Supporting Audiovisual Speech Processing}},
author = {H. C. Smith and R. M. O'Leary and J. E. Peelle},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.26.762007},
url = {https://www.biorxiv.org/content/10.64898/2026.10.26.762007v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Naturalistic Movie Viewing Reveals a Distributed Network Supporting Audiovisual Speech Processing
AU - H. C. Smith
AU - R. M. O'Leary
AU - J. E. Peelle
PY - 2026
DA - 2026/10/26
PB - bioRxiv
DO - 10.64898/2026.10.26.762007
UR - https://www.biorxiv.org/content/10.64898/2026.10.26.762007v1
AB - Traditional paradigms in cognitive neuroscience often fail to capture the richness of real-world experiences. To bridge this gap, researchers increasingly use naturalistic stimuli such as movies, which provide complex, multisensory input. Speech perception is a prime example of such complexity: it is inherently multisensory, involving the integration of auditory cues, lip movements, and facial expressions. Prior studies suggest that the left posterior superior temporal sulcus (pSTS) and superior temporal gyrus (STG) play a critical role in this integration. However, most studies of audiovisual speech have relied on simplified or artificial stimuli, limiting ecological validity. To address these gaps, we analyzed fMRI data from human participants watching two full-length films (500 Days of Summer, Citizenfour) to assess the consistency of multisensory speech processes across distinct genres and narratives. We manually annotated word onset times and modality (audiovisual vs. auditory-only). Across two movies and 38 participants we observed consistent speech-related activation in the pSTS and STG across speech types. Compared to auditory-only speech, audiovisual speech recruited a broad left-lateralized network including the angular gyrus, posterior temporal cortex, temporal pole, and inferior frontal gyrus, along with the right cerebellum. These findings were also consistently present in single-subject data. Importantly, most of the regions observed in our analyses have been implicated in multisensory function in the nonhuman primate literature despite receiving relatively little focus in human neuroimaging. Together our results demonstrate that speech processing during naturalistic viewing recruits a distributed network extending substantially beyond regions traditionally emphasized in audiovisual speech research.
ER -