Robust multimodal expert learning for sentiment analysis with missing modalities
- 1School of Artificial Intelligence, Xidian University
- 2School of Journalism and Communication, Northwest University
Abstract
Multimodal sentiment analysis (MSA) aims to infer sentiment by jointly modeling textual, acoustic, and visual information. Although existing multimodal fusion methods achieve strong performance when all modalities are available, their reliability often decreases when one or more modality streams are absent or unevenly informative. To address this problem, this paper proposes RoME: Robust multimodal expert learning for sentiment analysis with missing modalities, a two-stage robust multimodal embedding framework for sentiment analysis under incomplete modality conditions. In the first stage, RoME refines unimodal representations using modality-specific experts with soft routing, preserving modality-dependent structure under partial observations. In the second stage, RoME constructs adaptive multimodal representations through a Cross-Modal Interaction Transformer for inter-modality modeling, a Modality Reliability Re-Weighting mechanism to regulate modality contributions, a Residual Multimodal Corrector to compensate for fusion errors, and a Cross-Modal Denoising Autoencoder that provides reconstruction-driven regularization. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that RoME achieves strong performance across full-modality, partial-modality, and single-modality settings. Ablation studies, modality contribution analysis, and cross-validation results further confirm the effectiveness and stability of the proposed framework under controlled missing-modality conditions. Our code is available at https://github.com/atifmallo/RoME .
Related
- Machine Learning and Deep Learning for Early Disease Detection Across Modalities: A Comprehensive Review of Methods, Applications, and Clinical Translation
- A multi-task learning approach for consumer sentiment classification and emotion detection via behavioral data
- Teaching an ECG model where to look does not make it more robust to missing leads: a pre-registered comparison of guideline-based attention guidance with random lead masking
- A novel hybrid and ensemble deep learning model for enhancing sentiment analysis in Persian context
- Exploring Deep Transfer Learning for Medical Image Processing and Analysis: A Comprehensive Analysis across Modalities
BibTeX
@article{hussain2026robust,
title = {{Robust multimodal expert learning for sentiment analysis with missing modalities}},
author = {Atif Hussain and Yu Gu and Iftekharul Islam and Aamir Ali and He Zhang and Chilato Chilato},
journal = {Scientific Reports},
year = {2026},
month = jul,
doi = {10.1038/s41598-026-60943-7},
url = {https://doi.org/10.1038/s41598-026-60943-7},
note = {CC BY},
}
RIS
TY - JOUR
TI - Robust multimodal expert learning for sentiment analysis with missing modalities
AU - Atif Hussain
AU - Yu Gu
AU - Iftekharul Islam
AU - Aamir Ali
AU - He Zhang
AU - Chilato Chilato
PY - 2026
DA - 2026/07/01
JO - Scientific Reports
DO - 10.1038/s41598-026-60943-7
UR - https://doi.org/10.1038/s41598-026-60943-7
AB - Multimodal sentiment analysis (MSA) aims to infer sentiment by jointly modeling textual, acoustic, and visual information. Although existing multimodal fusion methods achieve strong performance when all modalities are available, their reliability often decreases when one or more modality streams are absent or unevenly informative. To address this problem, this paper proposes RoME: Robust multimodal expert learning for sentiment analysis with missing modalities, a two-stage robust multimodal embedding framework for sentiment analysis under incomplete modality conditions. In the first stage, RoME refines unimodal representations using modality-specific experts with soft routing, preserving modality-dependent structure under partial observations. In the second stage, RoME constructs adaptive multimodal representations through a Cross-Modal Interaction Transformer for inter-modality modeling, a Modality Reliability Re-Weighting mechanism to regulate modality contributions, a Residual Multimodal Corrector to compensate for fusion errors, and a Cross-Modal Denoising Autoencoder that provides reconstruction-driven regularization. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that RoME achieves strong performance across full-modality, partial-modality, and single-modality settings. Ablation studies, modality contribution analysis, and cross-validation results further confirm the effectiveness and stability of the proposed framework under controlled missing-modality conditions. Our code is available at https://github.com/atifmallo/RoME .
SN - 2045-2322
ER -