Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback.
- 1Jinshen College, Nanjing Audit University, Nanjing, China.
Abstract
Current English language learning systems are often limited by single-dimensional learning-state perception, insufficient modeling of affective-cognitive characteristics, and delayed or inflexible feedback regulation. To address these limitations, this study proposes an affective-cognitive gated cross-modal transformer (ACG-CMT) for embodied multimodal human-computer interaction in English language learning. The framework integrates learners' speech, gestures, and affective responses as multimodal inputs. The speech encoder learns low-level acoustic-prosodic representations from Mel-spectral, energy, pitch/prosodic, and temporal patterns rather than receiving rubric-derived target scores; a spatiotemporal motion encoder captures gesture and posture dynamics; and an affective encoder models facial-action and vocal-affect cues. A cross-modal transformer captures temporal dependencies and contextual interactions among linguistic output, embodied behavior, and affective fluctuations. During fusion, an affective-cognitive gate adjusts modality-specific contributions using latent affective/cognitive context and pre-decision task context available at inference time, without access to ground-truth target labels. A multi-task decision module then supports learning-state recognition, affective-cognitive load estimation, and feedback-policy matching. ACG-CMT achieves an accuracy of 0.892, a macro-F1 of 0.884, an AUC of 0.941, and a difficult-state F1 of 0.877 for learning-state recognition; R2 = 0.836, RMSE = 0.084, and MAE = 0.068 for the operational affective-cognitive load reference index; and a feedback-policy matching rate of 0.861 with an average latency of 438 ms. These results indicate that the framework can integrate multimodal behavioral information for learner-state estimation and adaptive feedback, while the internal gating patterns are interpreted as model-level associations rather than causal psychological mechanisms.
Related
- A study on kindergarten teacher–child interaction quality based on a multimodal emotion recognition model integrating speech and text
- Collaborative reasoning framework integrating multimodal large language models and knowledge graphs for dragon and lion dance technical action semantic understanding
- EBITH: A Smartphone-Based Multimodal Accessibility System for Blind and Deaf Users Using Computer Vision and Sign Language Recognition
- Smartphone-Native Bidirectional Olfactory Multimodal Large Language Models: A Unified Framework for Smell Sensing, Semantic Reasoning, Scent Generation, and Closed-Loop Olfactory Interaction
- Smartphone-Native Bidirectional Pain-Related Multimodal Large Language Models: A Unified Framework for Pain Sensing, Contextual Reasoning, Safe Sensory Feedback, and Adaptive Interaction
BibTeX
@article{tang2026embodied,
title = {{Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback.}},
author = {Yao Tang},
journal = {Frontiers in neurorobotics},
year = {2026},
month = oct,
doi = {10.3389/fnbot.2026.1920739},
url = {https://europepmc.org/article/MED/42953040},
note = {CC BY},
}
RIS
TY - JOUR
TI - Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback.
AU - Yao Tang
PY - 2026
DA - 2026/10/14
JO - Frontiers in neurorobotics
DO - 10.3389/fnbot.2026.1920739
UR - https://europepmc.org/article/MED/42953040
AB - Current English language learning systems are often limited by single-dimensional learning-state perception, insufficient modeling of affective-cognitive characteristics, and delayed or inflexible feedback regulation. To address these limitations, this study proposes an affective-cognitive gated cross-modal transformer (ACG-CMT) for embodied multimodal human-computer interaction in English language learning. The framework integrates learners' speech, gestures, and affective responses as multimodal inputs. The speech encoder learns low-level acoustic-prosodic representations from Mel-spectral, energy, pitch/prosodic, and temporal patterns rather than receiving rubric-derived target scores; a spatiotemporal motion encoder captures gesture and posture dynamics; and an affective encoder models facial-action and vocal-affect cues. A cross-modal transformer captures temporal dependencies and contextual interactions among linguistic output, embodied behavior, and affective fluctuations. During fusion, an affective-cognitive gate adjusts modality-specific contributions using latent affective/cognitive context and pre-decision task context available at inference time, without access to ground-truth target labels. A multi-task decision module then supports learning-state recognition, affective-cognitive load estimation, and feedback-policy matching. ACG-CMT achieves an accuracy of 0.892, a macro-F1 of 0.884, an AUC of 0.941, and a difficult-state F1 of 0.877 for learning-state recognition; R2 = 0.836, RMSE = 0.084, and MAE = 0.068 for the operational affective-cognitive load reference index; and a feedback-policy matching rate of 0.861 with an average latency of 438 ms. These results indicate that the framework can integrate multimodal behavioral information for learner-state estimation and adaptive feedback, while the internal gating patterns are interpreted as model-level associations rather than causal psychological mechanisms.
SN - 1662-5218
ER -