{
  "abstract": "Current English language learning systems are often limited by single-dimensional learning-state perception, insufficient modeling of affective-cognitive characteristics, and delayed or inflexible feedback regulation. To address these limitations, this study proposes an affective-cognitive gated cross-modal transformer (ACG-CMT) for embodied multimodal human-computer interaction in English language learning. The framework integrates learners' speech, gestures, and affective responses as multimodal inputs. The speech encoder learns low-level acoustic-prosodic representations from Mel-spectral, energy, pitch/prosodic, and temporal patterns rather than receiving rubric-derived target scores; a spatiotemporal motion encoder captures gesture and posture dynamics; and an affective encoder models facial-action and vocal-affect cues. A cross-modal transformer captures temporal dependencies and contextual interactions among linguistic output, embodied behavior, and affective fluctuations. During fusion, an affective-cognitive gate adjusts modality-specific contributions using latent affective/cognitive context and pre-decision task context available at inference time, without access to ground-truth target labels. A multi-task decision module then supports learning-state recognition, affective-cognitive load estimation, and feedback-policy matching. ACG-CMT achieves an accuracy of 0.892, a macro-F1 of 0.884, an AUC of 0.941, and a difficult-state F1 of 0.877 for learning-state recognition; R2 = 0.836, RMSE = 0.084, and MAE = 0.068 for the operational affective-cognitive load reference index; and a feedback-policy matching rate of 0.861 with an average latency of 438 ms. These results indicate that the framework can integrate multimodal behavioral information for learner-state estimation and adaptive feedback, while the internal gating patterns are interpreted as model-level associations rather than causal psychological mechanisms.",
  "abstract_shown": true,
  "affiliations": [
    "Jinshen College, Nanjing Audit University, Nanjing, China."
  ],
  "article_type": "research article",
  "arxiv": null,
  "author_total": 1,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Yao Tang"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/bh47hzb5ahyr2.bib",
  "cited_by": 0,
  "doi": "10.3389/fnbot.2026.1920739",
  "fulltext_url": "https://europepmc.org/articles/PMC13713760",
  "id": "bh47hzb5ahyr2",
  "issns": [
    "1662-5218"
  ],
  "landing_url": "https://europepmc.org/article/MED/42953040",
  "language": "eng",
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "heading",
      "Original Research"
    ],
    [
      "openalex",
      "1702|Intelligent Tutoring Systems and Adaptive Learning"
    ]
  ],
  "openalex": "W7221829073",
  "pdf_url": "https://europepmc.org/articles/PMC13713760?pdf=render",
  "pmcid": "PMC13713760",
  "pmid": "42953040",
  "published": "2026-10-14",
  "publisher": "Frontiers Media SA",
  "ris": "https://research.draconitic.com/paper/bh47hzb5ahyr2.ris",
  "sources": [
    {
      "first_seen": "2026-10-31T00:20:41Z",
      "native_id": "MED:42953040",
      "source": "europepmc",
      "url": "https://europepmc.org/article/MED/42953040"
    },
    {
      "first_seen": "2026-11-01T12:21:22Z",
      "native_id": "PMC13713760",
      "source": "pmc",
      "url": "https://doi.org/10.3389/fnbot.2026.1920739"
    }
  ],
  "subjects": [],
  "title": "Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback.",
  "title_text": "Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback.",
  "type": "peer-reviewed",
  "updated_at": "2026-11-01T12:21:22Z",
  "url": "https://research.draconitic.com/paper/bh47hzb5ahyr2",
  "venue": "Frontiers in neurorobotics",
  "versions": []
}