Brain-inspired predictive visual attention for spatiotemporal semantic understanding of complex film and television videos
- 1Nanjing Institute of Technology, Nanjing, Jiangsu, China
Abstract
Due to the unique characteristics of film and television, such as montage editing and narrative structure jumps, semantically salient targets frequently experience temporal feature drift, making it difficult for existing models to maintain stable semantic understanding output when keyframes are lost. Inspired by predictive coding mechanisms in neuroscience, this paper proposes a spatiotemporal joint representation framework driven by brain-like visual attention. A pixel-level saliency heatmap is constructed using eye-tracking data as a physiological-level supervisory signal. A Dual-Stream Spatio-Temporal Encoding Network (DS-STEN) is designed: the spatial stream uses ResNet to extract single-frame salient features, while the temporal stream uses 3D convolution to model the optical flow field and capture cross-frame trajectories. A Kalman filter-driven predictive attention module is introduced into the fusion layer to estimate the target’s position and feature distribution in the next frame. The prediction results are injected into the temporal stream as a priori mask to actively correct abnormal responses during drift. The corrected spatiotemporal features are aggregated with semantic context through a Transformer cross-frame self-attention layer and input into a semantic classification head to generate scene-level and event-level labels. End-to-end joint training is performed using a weighted loss of saliency prediction, trajectory consistency, and semantic classification. Experiments show that the saliency prediction correlation coefficient (CC) reaches 0.846 and the normalized scanpath saliency (NSS) reaches 2.47 on the AIM2024 benchmark. In the TVQA+ dataset target tracking task, the cross-frame tracking accuracy in unoccluded scenes reaches 95.3%. In the MovieNet dataset scene semantic classification task, the average accuracy for Dialogue scenes reaches 93.5%. This paper provides a new spatiotemporal modeling paradigm with a neuroscience basis for semantic understanding of complex film and television videos.
Related
- TCCA: An Inter-Layer Spatiotemporal Cross-Attention Network for IIoT Attack Traffic Detection
- MagGAT: Structure-Aware Spatiotemporal Graph Attention for Motor-Induced Magnetic Interference Compensation in Quadruped Robots
- Spatiotemporal Dynamics of Inter-Item Competition in Visual Working Memory: The Role of Internal Attention
- Dynamic heterogeneous graph attention network for spatiotemporal correlation modeling of mine microseismic events and intelligent rockburst risk level reasoning
- A neuro-inspired computational framework for AGI: predictive coding, active inference and free energy minimisation.
BibTeX
@article{han2026braininspired,
title = {{Brain-inspired predictive visual attention for spatiotemporal semantic understanding of complex film and television videos}},
author = {Peng Han},
journal = {Frontiers in Neurorobotics},
year = {2026},
month = oct,
doi = {10.3389/fnbot.2026.1934032},
url = {https://doi.org/10.3389/fnbot.2026.1934032},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Brain-inspired predictive visual attention for spatiotemporal semantic understanding of complex film and television videos
AU - Peng Han
PY - 2026
DA - 2026/10/22
JO - Frontiers in Neurorobotics
DO - 10.3389/fnbot.2026.1934032
UR - https://doi.org/10.3389/fnbot.2026.1934032
AB - Due to the unique characteristics of film and television, such as montage editing and narrative structure jumps, semantically salient targets frequently experience temporal feature drift, making it difficult for existing models to maintain stable semantic understanding output when keyframes are lost. Inspired by predictive coding mechanisms in neuroscience, this paper proposes a spatiotemporal joint representation framework driven by brain-like visual attention. A pixel-level saliency heatmap is constructed using eye-tracking data as a physiological-level supervisory signal. A Dual-Stream Spatio-Temporal Encoding Network (DS-STEN) is designed: the spatial stream uses ResNet to extract single-frame salient features, while the temporal stream uses 3D convolution to model the optical flow field and capture cross-frame trajectories. A Kalman filter-driven predictive attention module is introduced into the fusion layer to estimate the target’s position and feature distribution in the next frame. The prediction results are injected into the temporal stream as a priori mask to actively correct abnormal responses during drift. The corrected spatiotemporal features are aggregated with semantic context through a Transformer cross-frame self-attention layer and input into a semantic classification head to generate scene-level and event-level labels. End-to-end joint training is performed using a weighted loss of saliency prediction, trajectory consistency, and semantic classification. Experiments show that the saliency prediction correlation coefficient (CC) reaches 0.846 and the normalized scanpath saliency (NSS) reaches 2.47 on the AIM2024 benchmark. In the TVQA+ dataset target tracking task, the cross-frame tracking accuracy in unoccluded scenes reaches 95.3%. In the MovieNet dataset scene semantic classification task, the average accuracy for Dialogue scenes reaches 93.5%. This paper provides a new spatiotemporal modeling paradigm with a neuroscience basis for semantic understanding of complex film and television videos.
SN - 1662-5218
ER -