{
  "abstract": "Due to the unique characteristics of film and television, such as montage editing and narrative structure jumps, semantically salient targets frequently experience temporal feature drift, making it difficult for existing models to maintain stable semantic understanding output when keyframes are lost. Inspired by predictive coding mechanisms in neuroscience, this paper proposes a spatiotemporal joint representation framework driven by brain-like visual attention. A pixel-level saliency heatmap is constructed using eye-tracking data as a physiological-level supervisory signal. A Dual-Stream Spatio-Temporal Encoding Network (DS-STEN) is designed: the spatial stream uses ResNet to extract single-frame salient features, while the temporal stream uses 3D convolution to model the optical flow field and capture cross-frame trajectories. A Kalman filter-driven predictive attention module is introduced into the fusion layer to estimate the target’s position and feature distribution in the next frame. The prediction results are injected into the temporal stream as a priori mask to actively correct abnormal responses during drift. The corrected spatiotemporal features are aggregated with semantic context through a Transformer cross-frame self-attention layer and input into a semantic classification head to generate scene-level and event-level labels. End-to-end joint training is performed using a weighted loss of saliency prediction, trajectory consistency, and semantic classification. Experiments show that the saliency prediction correlation coefficient (CC) reaches 0.846 and the normalized scanpath saliency (NSS) reaches 2.47 on the AIM2024 benchmark. In the TVQA+ dataset target tracking task, the cross-frame tracking accuracy in unoccluded scenes reaches 95.3%. In the MovieNet dataset scene semantic classification task, the average accuracy for Dialogue scenes reaches 93.5%. This paper provides a new spatiotemporal modeling paradigm with a neuroscience basis for semantic understanding of complex film and television videos.",
  "abstract_shown": true,
  "affiliations": [
    "Nanjing Institute of Technology, Nanjing, Jiangsu, China"
  ],
  "article_type": "research article",
  "arxiv": null,
  "author_total": 1,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Peng Han"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/90zn3gmsps8hf.bib",
  "cited_by": 0,
  "doi": "10.3389/fnbot.2026.1934032",
  "fulltext_url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13735105/",
  "id": "90zn3gmsps8hf",
  "issns": [
    "1662-5218"
  ],
  "landing_url": "https://doi.org/10.3389/fnbot.2026.1934032",
  "language": "en",
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "heading",
      "Original Research"
    ],
    [
      "openalex",
      "1707|Visual Attention and Saliency Detection"
    ]
  ],
  "openalex": "W7223170941",
  "pdf_url": "https://www.frontiersin.org/journals/neurorobotics/articles/10.3389/fnbot.2026.1934032/pdf",
  "pmcid": "PMC13735105",
  "pmid": "42991078",
  "published": "2026-10-22",
  "publisher": "Frontiers Media SA",
  "ris": "https://research.draconitic.com/paper/90zn3gmsps8hf.ris",
  "sources": [
    {
      "first_seen": "2026-11-09T12:20:52Z",
      "native_id": "PMC13735105",
      "source": "pmc",
      "url": "https://doi.org/10.3389/fnbot.2026.1934032"
    }
  ],
  "subjects": [],
  "title": "Brain-inspired predictive visual attention for spatiotemporal semantic understanding of complex film and television videos",
  "title_text": "Brain-inspired predictive visual attention for spatiotemporal semantic understanding of complex film and television videos",
  "type": "peer-reviewed",
  "updated_at": "2026-11-09T12:20:52Z",
  "url": "https://research.draconitic.com/paper/90zn3gmsps8hf",
  "venue": "Frontiers in Neurorobotics",
  "versions": []
}