Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex
Abstract
Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the majority of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding models address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fitting this model to fMRI responses to short video clips. We compare joint spatiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides interpretable, stimulus-specific attention maps that dynamically follow moving objects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category-selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception.
Related
- Spatiotemporal encoding signatures of covert speech revealed by source-localized EEG and fMRI
- A Single-cell Spatiotemporal Atlas Encoding Age-dependent Evolution of the Ischemic Penumbra
- A non-local multi-head spatiotemporal attention LSTM for vehicle trajectory prediction
- Artificial microRNAs against hypoxia-responsive small protein encoding genes reveal HRSP41 and 42 as regulators of development and stress tolerance
- Vision-language encoding models reveal an image-computable food-quality dimension in human occipitotemporal cortex
BibTeX
@misc{inabathini2026crossattention,
title = {{Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex}},
author = {Iishaan Inabathini and Margaret M. Henderson},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.36366},
eprint = {2609.36366},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.36366},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex
AU - Iishaan Inabathini
AU - Margaret M. Henderson
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.36366
UR - https://arxiv.org/abs/2609.36366
AB - Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the majority of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding models address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fitting this model to fMRI responses to short video clips. We compare joint spatiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides interpretable, stimulus-specific attention maps that dynamically follow moving objects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category-selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception.
ER -