Neural tracking of surprisal and semantic distance in naturalistic movie viewing
- 1Northeastern University
Abstract
Understanding speech requires listeners to integrate incoming input with prior linguistic and thematic knowledge to access meaning, a task greatly aided by prediction. Surprisal and related phenomena (e.g., next word prediction) tend to be associated with broad activation of language regions during listening. A major challenge for interpreting the neural correlates of surprisal is that lexical expectancies are constrained both by syntactic and semantic information. In contrast, semantic distance yields more of a pure metric of relatedness between one language constituent (e.g., word, phrase, n-gram) and another within a high-dimensional semantic space. We contrasted surprisal and semantic distance in running discourse during naturalistic audiovisual language comprehension. We analyzed fMRI scans from 20 English-speaking adults who viewed the full-length film 500 Days of Summer. Our focus was on brain sensitivity to either surprisal or semantic distance as language unfolded word-by-word. To this end, we implemented a series of hierarchical voxelwise encoding models. After accounting for low-level auditory and visual features, the addition of either surprisal or semantic distance improved prediction in bilateral superior temporal gyrus (STG), left inferior frontal gyrus, and right cerebellum. Critically, both semantic distance and surprisal explained unique variation in bilateral temporal cortex after accounting for the other. These results support the notion that the brain tracks both broad probabilistic prediction as well as semantic integration. More generally, these findings extend previous work on auditory-only speech by demonstrating that surprisal and semantic distance predict meaningful neural variation even within a rich audiovisual context.
Related
- From Exposure to Experience: Measuring Gaze-Based Visual Experience with Faces During Naturalistic Viewing
- Semantic Update as a Predictor of Reading Time: Moving Beyond Word-Level Surprisal
- Individual differences in eye movements dynamics of free viewing conscious and unconscious naturalistic scenes
- rpSTS Stimulation Disrupts Shared Neural Responses to Social Content during Naturalistic Viewing
- Naturalistic Movie Viewing Reveals a Distributed Network Supporting Audiovisual Speech Processing
BibTeX
@misc{oleary2026neural,
title = {{Neural tracking of surprisal and semantic distance in naturalistic movie viewing}},
author = {R. M. O'Leary and H. C. Smith and E. B. Myers and J. Reilly and J. E. Peelle},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.09.749230},
url = {https://www.biorxiv.org/content/10.64898/2026.09.09.749230v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Neural tracking of surprisal and semantic distance in naturalistic movie viewing
AU - R. M. O'Leary
AU - H. C. Smith
AU - E. B. Myers
AU - J. Reilly
AU - J. E. Peelle
PY - 2026
DA - 2026/09/09
PB - bioRxiv
DO - 10.64898/2026.09.09.749230
UR - https://www.biorxiv.org/content/10.64898/2026.09.09.749230v1
AB - Understanding speech requires listeners to integrate incoming input with prior linguistic and thematic knowledge to access meaning, a task greatly aided by prediction. Surprisal and related phenomena (e.g., next word prediction) tend to be associated with broad activation of language regions during listening. A major challenge for interpreting the neural correlates of surprisal is that lexical expectancies are constrained both by syntactic and semantic information. In contrast, semantic distance yields more of a pure metric of relatedness between one language constituent (e.g., word, phrase, n-gram) and another within a high-dimensional semantic space. We contrasted surprisal and semantic distance in running discourse during naturalistic audiovisual language comprehension. We analyzed fMRI scans from 20 English-speaking adults who viewed the full-length film 500 Days of Summer. Our focus was on brain sensitivity to either surprisal or semantic distance as language unfolded word-by-word. To this end, we implemented a series of hierarchical voxelwise encoding models. After accounting for low-level auditory and visual features, the addition of either surprisal or semantic distance improved prediction in bilateral superior temporal gyrus (STG), left inferior frontal gyrus, and right cerebellum. Critically, both semantic distance and surprisal explained unique variation in bilateral temporal cortex after accounting for the other. These results support the notion that the brain tracks both broad probabilistic prediction as well as semantic integration. More generally, these findings extend previous work on auditory-only speech by demonstrating that surprisal and semantic distance predict meaningful neural variation even within a rich audiovisual context.
ER -