Hierarchical multimodal semantic fusion for few-shot reader action recognition in reading environments
Abstract
Understanding reader behavior in real-world reading environments is important for data-driven content services and intelligent publishing analytics. However, most action recognition methods target motion-dominant scenarios and struggle under few-shot conditions to capture the subtle visual cues and hierarchical semantics of reading-related behaviors. To address this problem, we construct the Reader Action Recognition Dataset (RARS), containing 12 action classes and 1,728 video clips, and propose a Hierarchical Semantic Fusion Network (HSFN) for few-shot reader action recognition. HSFN maintains a shared semantic structure spanning the global, patch, and frame levels across hierarchical visual and textual representations, granularity-consistent multimodal fusion, hyperbolic multi-prototype alignment, and dual-path video-to-video and video-to-text matching. Counterfactual supervision and adaptive-margin multi-grained contrastive learning further refine the boundaries between semantically similar actions. Experiments on RARS and the public SSv2-Small benchmark validate the effectiveness of HSFN under 5-way 1-shot and 5-way 5-shot settings. Ablation studies, attention-guided occlusion, and failure-case analysis further support the proposed components and clarify HSFN’s recognition characteristics for semantically similar actions. Overall, although HSFN remains limited when distinctions between semantically similar actions depend on transient temporal cues or small interacting objects, it provides a viable few-shot solution for fine-grained reader action recognition in intelligent reading environments.
Related
- Visual–Semantic Processing and Reading Comprehension in Chinese Deaf and Hard-of-Hearing Students: Roles of Updating, Semantic Radical Awareness, and Daily Sign Language Use
- Semantic Update as a Predictor of Reading Time: Moving Beyond Word-Level Surprisal
- Vision-semantic alignment with hierarchical consistency constraints for multi-attribute museum artifact classification
- Bidirectional cross-scale detail-semantic collaborative network (BCDS-Net) for vision-based waste recognition
- Collaborative reasoning framework integrating multimodal large language models and knowledge graphs for dragon and lion dance technical action semantic understanding
BibTeX
@article{wang2026hierarchical,
title = {{Hierarchical multimodal semantic fusion for few-shot reader action recognition in reading environments}},
author = {Zhaohua Wang and Jiawen Li and Yuqian Wang and Yirong Luo and Jiayi Yang and Zheng Li and Xinong En and Yanping Du},
journal = {Scientific Reports},
year = {2026},
month = sep,
doi = {10.1038/s41598-026-71846-y},
url = {https://www.nature.com/articles/s41598-026-71846-y},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Hierarchical multimodal semantic fusion for few-shot reader action recognition in reading environments
AU - Zhaohua Wang
AU - Jiawen Li
AU - Yuqian Wang
AU - Yirong Luo
AU - Jiayi Yang
AU - Zheng Li
AU - Xinong En
AU - Yanping Du
PY - 2026
DA - 2026/09/30
JO - Scientific Reports
DO - 10.1038/s41598-026-71846-y
UR - https://www.nature.com/articles/s41598-026-71846-y
AB - Understanding reader behavior in real-world reading environments is important for data-driven content services and intelligent publishing analytics. However, most action recognition methods target motion-dominant scenarios and struggle under few-shot conditions to capture the subtle visual cues and hierarchical semantics of reading-related behaviors. To address this problem, we construct the Reader Action Recognition Dataset (RARS), containing 12 action classes and 1,728 video clips, and propose a Hierarchical Semantic Fusion Network (HSFN) for few-shot reader action recognition. HSFN maintains a shared semantic structure spanning the global, patch, and frame levels across hierarchical visual and textual representations, granularity-consistent multimodal fusion, hyperbolic multi-prototype alignment, and dual-path video-to-video and video-to-text matching. Counterfactual supervision and adaptive-margin multi-grained contrastive learning further refine the boundaries between semantically similar actions. Experiments on RARS and the public SSv2-Small benchmark validate the effectiveness of HSFN under 5-way 1-shot and 5-way 5-shot settings. Ablation studies, attention-guided occlusion, and failure-case analysis further support the proposed components and clarify HSFN’s recognition characteristics for semantically similar actions. Overall, although HSFN remains limited when distinctions between semantically similar actions depend on transient temporal cues or small interacting objects, it provides a viable few-shot solution for fine-grained reader action recognition in intelligent reading environments.
SN - 2045-2322
ER -