SCL: Semantic Coherence Learning for Video Question Answering
Abstract
Video Question Answering (VideoQA) requires clarifying the true question intent and understanding dynamic video content to predict the answer. However, most existing methods overlook the temporal semantic exploration of fine-grained clues across frames, and neglect the interrogative properties of question text, leading to suboptimal grounding of video frames. To address these limitations, in this paper, we propose a novel Semantic Coherence Learning (SCL) framework, which explicitly models the fine-grained spatio-temporal coherence of critical visual regions and progressively mines multi-modal semantic clues aligned with the reasoning intent. Specifically, we design a spatio-temporal clue reasoning module to adaptively model the fine-grained coherent transition semantics of critical regions across temporal frames while suppressing irrelevant noisy regions within frames. Additionally, we propose a multi-modal clue reasoning module to progressively mine multi-modal coherent semantics associated with reasoning intent, which iteratively grounds critical frames in videos based on refined question semantics and mines critical phrases in questions based on refined video semantics. Finally, the answer is derived through an answer decoder with intra and inter-sample contrastive strategies. Extensive experimental results on six widely used benchmarks verify our effectiveness and superiority. The experimental source codes are available at https://github.com/XizeWu/SCL.
Related
- Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models
- Calibrated Bayesian Selection of Schema Subgraphs for Database Question Answering
- Memory in the Sky: Low-Altitude Question Answering with Multi-Agent Memory Aggregation
- Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission
- Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation
BibTeX
@article{wu2026semantic,
title = {{SCL: Semantic Coherence Learning for Video Question Answering}},
author = {Xize Wu and Zheng Wang and Jiasong Wu and Lei Zhu and Heng Tao Shen},
journal = {IEEE Transactions on Multimedia},
year = {2026},
month = sep,
doi = {10.1109/tmm.2026.3668493},
url = {https://univ-rennes.hal.science/hal-05682134v1},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - SCL: Semantic Coherence Learning for Video Question Answering
AU - Xize Wu
AU - Zheng Wang
AU - Jiasong Wu
AU - Lei Zhu
AU - Heng Tao Shen
PY - 2026
DA - 2026/09/17
JO - IEEE Transactions on Multimedia
DO - 10.1109/tmm.2026.3668493
UR - https://univ-rennes.hal.science/hal-05682134v1
AB - Video Question Answering (VideoQA) requires clarifying the true question intent and understanding dynamic video content to predict the answer. However, most existing methods overlook the temporal semantic exploration of fine-grained clues across frames, and neglect the interrogative properties of question text, leading to suboptimal grounding of video frames. To address these limitations, in this paper, we propose a novel Semantic Coherence Learning (SCL) framework, which explicitly models the fine-grained spatio-temporal coherence of critical visual regions and progressively mines multi-modal semantic clues aligned with the reasoning intent. Specifically, we design a spatio-temporal clue reasoning module to adaptively model the fine-grained coherent transition semantics of critical regions across temporal frames while suppressing irrelevant noisy regions within frames. Additionally, we propose a multi-modal clue reasoning module to progressively mine multi-modal coherent semantics associated with reasoning intent, which iteratively grounds critical frames in videos based on refined question semantics and mines critical phrases in questions based on refined video semantics. Finally, the answer is derived through an answer decoder with intra and inter-sample contrastive strategies. Extensive experimental results on six widely used benchmarks verify our effectiveness and superiority. The experimental source codes are available at https://github.com/XizeWu/SCL.
SN - 1520-9210
ER -