Dynamic hypernode injection and graph attention fusion for multimodal sentiment analysis of behavioral signals
- 1College of Intelligent Systems Science and Engineering, Hubei Minzu University, Enshi, China
Abstract
Language use, vocal behavior, and facial activity provide complementary indicators of affective state, but multimodal models can be affected by noisy temporal observations and insufficient global guidance before cross-modal interaction. We developed a framework combining bidirectional gated recurrent unit encoders, attention pooling, dynamic hypernode injection, and graph attention fusion. Textual, acoustic, and visual sequences were mapped into a shared latent space and compressed into modality-level representations. A sample-dependent hypernode and a learnable static prior were then injected through gated residual connections before graph propagation. The model was evaluated on CMU-MOSI and CMU-MOSEI using five random seeds and validation-MAE checkpoint selection. On CMU-MOSI, the model obtained an MAE of 0.7290±0.0099, a correlation of 0.7880±0.0076, and an Acc-7 of 0.4561±0.0160. On CMU-MOSEI, the corresponding results were 0.5541±0.0053, 0.7551±0.0058, and 0.5269±0.0064. A five-run comparison of pre-GAT, post-GAT, and no injection showed that pre-GAT injection provided the strongest overall regression-oriented balance, although post-GAT was slightly higher on MOSEI Acc-7. Gate, graph-attention, representation, and balanced case diagnostics showed that the injected prior was sample dependent and modality selective. These findings support pre-propagation hypernode injection under complete, word-aligned tri-modal inputs, without establishing universal superiority or robustness to missing and corrupted modalities.
Related
- Sentiment classification optimization based on improved Senti-BERT and BiLSTM-Attention
- Behavioral signatures suggest distinct modes of suppressing irrelevant information during tactile temporal attention in human participants
- CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments
- MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus
- TransBind2: Improving Transcription Factor-DNA Binding Prediction with Multimodal Data and Bidirectional Cross Attention
BibTeX
@article{cao2026dynamic,
title = {{Dynamic hypernode injection and graph attention fusion for multimodal sentiment analysis of behavioral signals}},
author = {Wenfei Cao and Jian Xu and Yinghao Li and Huizi Yan and Zhuowei Hu and Wenxu Chen},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0359425},
url = {https://doi.org/10.1371/journal.pone.0359425},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Dynamic hypernode injection and graph attention fusion for multimodal sentiment analysis of behavioral signals
AU - Wenfei Cao
AU - Jian Xu
AU - Yinghao Li
AU - Huizi Yan
AU - Zhuowei Hu
AU - Wenxu Chen
PY - 2026
DA - 2026/09/28
JO - PLOS One
DO - 10.1371/journal.pone.0359425
UR - https://doi.org/10.1371/journal.pone.0359425
AB - Language use, vocal behavior, and facial activity provide complementary indicators of affective state, but multimodal models can be affected by noisy temporal observations and insufficient global guidance before cross-modal interaction. We developed a framework combining bidirectional gated recurrent unit encoders, attention pooling, dynamic hypernode injection, and graph attention fusion. Textual, acoustic, and visual sequences were mapped into a shared latent space and compressed into modality-level representations. A sample-dependent hypernode and a learnable static prior were then injected through gated residual connections before graph propagation. The model was evaluated on CMU-MOSI and CMU-MOSEI using five random seeds and validation-MAE checkpoint selection. On CMU-MOSI, the model obtained an MAE of 0.7290±0.0099, a correlation of 0.7880±0.0076, and an Acc-7 of 0.4561±0.0160. On CMU-MOSEI, the corresponding results were 0.5541±0.0053, 0.7551±0.0058, and 0.5269±0.0064. A five-run comparison of pre-GAT, post-GAT, and no injection showed that pre-GAT injection provided the strongest overall regression-oriented balance, although post-GAT was slightly higher on MOSEI Acc-7. Gate, graph-attention, representation, and balanced case diagnostics showed that the injected prior was sample dependent and modality selective. These findings support pre-propagation hypernode injection under complete, word-aligned tri-modal inputs, without establishing universal superiority or robustness to missing and corrupted modalities.
SN - 1932-6203
ER -