Multimodal deep learning with particle swarm guided feature refinement for automated pain detection
Abstract
Pain assessment remains a significant challenge in clinical practice because conventional self-reporting methods are subjective and unreliable for non-verbal, cognitively impaired, and critically ill patients. Existing automated pain detection approaches primarily rely on single-modality data or limited multimodal integration, resulting in inadequate feature representation, feature redundancy, and reduced classification performance. To address these limitations, this study proposes a multimodal deep learning framework that integrates physiological signals, facial images, magnetic resonance imaging (MRI), and clinical text for objective pain detection. The physiological branch employs a hybrid architecture comprising a Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM) network, Bidirectional Gated Recurrent Unit (BiGRU), and Deep Neural Network (DNN) to learn hierarchical spatial-temporal representations. The facial branch combines EfficientNet and Residual Network (ResNet) to extract discriminative facial features, while a CNN-based architecture processes MRI images. Clinical narratives are encoded using Bidirectional Encoder Representations from Transformers (BERT) and BioBERT to capture contextual semantic information. An Adaptive Feature Refinement Layer (AFRL) enhances modality-specific feature learning, followed by Particle Swarm Optimisation (PSO) for feature selection and a novel PSO-Guided Feature Refinement Layer (PFRL) for adaptive feature weighting prior to multimodal fusion. We implement the framework in TensorFlow and PyTorch and evaluate it on the BioVid, UNBC-McMaster, and Trigeminal Neuralgia MRI datasets. Experimental results achieve 96.2% accuracy, outperforming Support Vector Machine (82.4%), K-Nearest Neighbors (79.6%), Bidirectional Long Short-Term Memory (88.9%), and AdaBoost (90.7%). The proposed framework provides a robust, scalable, and clinically applicable solution for automated multimodal pain detection and improved clinical decision support.
Related
- MULTIMODAL EEG WORLD MODEL: SELF-SUPERVISED LATENT TRANSITION LEARNING FOR WEARABLE EEG SEIZURE DETECTION
- OsteoNet-X: A deep residual learning framework for automated osteoporosis detection from knee X-ray images
- Automated detection of focal cortical dysplasia in epilepsy using machine learning on structural MRI
- HemaViT: transformer-based deep learning for automated non-invasive anemia detection using conjunctival imaging
- An interpretable multimodal deep learning model for physiological stress detection
BibTeX
@article{gasmi2026multimodal,
title = {{Multimodal deep learning with particle swarm guided feature refinement for automated pain detection}},
author = {Karim Gasmi and Sarah Almaghrabi and Ibrahim Rizqallah Alzahrani and Adil Mahmoud Mohamed Mahmoud and Alameen Abdalrahman Eltoum and Abdullah Sawdi Alenzi and Osama R. Shahin and Ahmed I. Taloba and Fatma Al Zahraa Hamed Abdel Hameed},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-75973-4},
url = {https://www.nature.com/articles/s41598-026-75973-4},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Multimodal deep learning with particle swarm guided feature refinement for automated pain detection
AU - Karim Gasmi
AU - Sarah Almaghrabi
AU - Ibrahim Rizqallah Alzahrani
AU - Adil Mahmoud Mohamed Mahmoud
AU - Alameen Abdalrahman Eltoum
AU - Abdullah Sawdi Alenzi
AU - Osama R. Shahin
AU - Ahmed I. Taloba
AU - Fatma Al Zahraa Hamed Abdel Hameed
PY - 2026
DA - 2026/10/14
JO - Scientific Reports
DO - 10.1038/s41598-026-75973-4
UR - https://www.nature.com/articles/s41598-026-75973-4
AB - Pain assessment remains a significant challenge in clinical practice because conventional self-reporting methods are subjective and unreliable for non-verbal, cognitively impaired, and critically ill patients. Existing automated pain detection approaches primarily rely on single-modality data or limited multimodal integration, resulting in inadequate feature representation, feature redundancy, and reduced classification performance. To address these limitations, this study proposes a multimodal deep learning framework that integrates physiological signals, facial images, magnetic resonance imaging (MRI), and clinical text for objective pain detection. The physiological branch employs a hybrid architecture comprising a Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM) network, Bidirectional Gated Recurrent Unit (BiGRU), and Deep Neural Network (DNN) to learn hierarchical spatial-temporal representations. The facial branch combines EfficientNet and Residual Network (ResNet) to extract discriminative facial features, while a CNN-based architecture processes MRI images. Clinical narratives are encoded using Bidirectional Encoder Representations from Transformers (BERT) and BioBERT to capture contextual semantic information. An Adaptive Feature Refinement Layer (AFRL) enhances modality-specific feature learning, followed by Particle Swarm Optimisation (PSO) for feature selection and a novel PSO-Guided Feature Refinement Layer (PFRL) for adaptive feature weighting prior to multimodal fusion. We implement the framework in TensorFlow and PyTorch and evaluate it on the BioVid, UNBC-McMaster, and Trigeminal Neuralgia MRI datasets. Experimental results achieve 96.2% accuracy, outperforming Support Vector Machine (82.4%), K-Nearest Neighbors (79.6%), Bidirectional Long Short-Term Memory (88.9%), and AdaBoost (90.7%). The proposed framework provides a robust, scalable, and clinically applicable solution for automated multimodal pain detection and improved clinical decision support.
SN - 2045-2322
ER -