Skip to content
DraconiticResearch
Search Sources API
194,161 studies31 sourcesUpdated 43 min ago
Peer-reviewedCC BYNature Neuroscience

Simultaneous speech and gesture decoding for multimodal communication in paralysis

  • Samantha Brosler1,2,3
  • Jessie R. Liu1,3
  • Alexander B. Silva1,2,3
  • Irina P Hallinan1
  • Cady M Kurtz-Miott1
  • Jonah Dunkel Wilker1
  • Adelyn Tu-Chan4
  • Karunesh Ganguly4,3
  • Edward F. Chang5,6,7
7 affiliations
  1. 1Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
  2. 2University of California, Berkeley-University of California, San Francisco, Graduate Program in Bioengineering, Berkeley, CA, USA
  3. 3Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA
  4. 4Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
  5. 5Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA. [email protected]
  6. 6University of California, Berkeley-University of California, San Francisco, Graduate Program in Bioengineering, Berkeley, CA, USA. [email protected]
  7. 7Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA. [email protected]
Open at sourcePDFDOICite

Abstract

Stroke and neurodegenerative diseases can impair speech and nonverbal gestures, limiting natural communication. Brain-computer interfaces (BCIs) aim to restore these functions by translating neural activity into commands for external devices, although prior work has primarily focused on decoding speech or gestures in isolation. Here we show that neural signals recorded with a single high-density electrocorticography implant can support simultaneous decoding of speech and gestures in people with paralysis. We first show that isolated upper-limb and orofacial movements can be reliably decoded among three participants. Using parallel speech and gesture decoders, we then enabled participants to control a personalized virtual avatar by attempting speech and gestures simultaneously or in isolation. Training models on both isolated and simultaneous data improved performance across behavioral contexts. These findings demonstrate that one cortical implant can support multi-effector control and provide a step toward BCIs that enable more natural communication for people with paralysis.

Related

  1. Intact-to-Amputee Transfer in Surface-EMG Gesture Decoding: Training Source and Calibration Budget
    Jethro Odeyemi, W. J. Zhang
    arXivPreprintCC BY 4.0Full textNeurosciencearXiv
  2. Simultaneous multi-task assessment of five neurological language disorders via cross-corpus multimodal learning
    Byounghwa Lee, Hwa Jeon Song, Byung Ok Kang
    Scientific ReportsPeer-reviewedCC BY-NC-ND 4.0Full textNeuroscienceCrossref
  3. Multimodal Plant–Animal Communication: Signaling Mechanisms, Coevolution, and Environmental Disruption
    Hiba A. Jasim, Marwa Y. Khudair, Sura A. Jasim +1
    Plants (Basel, Switzerland)Peer-reviewedCC BYFull textPlant biologyEurope PMC
  4. Multimodal Physiological Decoding Reveals Individualized Arousal Dynamics in Closed-Loop Neurofeedback
    Anirudh Natarajan, Paul Sajda
    arXivPreprintCC BY 4.0Full textNeurosciencearXiv
  5. Real-time gesture mapping system for a 15-DOF palm-scale bionic manipulator
    Qunting Chen, Xinhui Chen, Geng Chen
    PLOS OnePeer-reviewedCC BY 4.0Full textSystems & synthetic biologyPhysiologyPLOS

Cite

BibTeXRIS
BibTeX
@article{brosler2026simultaneous,
  title = {{Simultaneous speech and gesture decoding for multimodal communication in paralysis}},
  author = {Samantha Brosler and Jessie R. Liu and Alexander B. Silva and Irina P Hallinan and Cady M Kurtz-Miott and Jonah Dunkel Wilker and Adelyn Tu-Chan and Karunesh Ganguly and Edward F. Chang},
  journal = {Nature Neuroscience},
  year = {2026},
  month = sep,
  doi = {10.1038/s41593-026-02446-2},
  url = {https://doi.org/10.1038/s41593-026-02446-2},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - Simultaneous speech and gesture decoding for multimodal communication in paralysis
AU  - Samantha Brosler
AU  - Jessie R. Liu
AU  - Alexander B. Silva
AU  - Irina P Hallinan
AU  - Cady M Kurtz-Miott
AU  - Jonah Dunkel Wilker
AU  - Adelyn Tu-Chan
AU  - Karunesh Ganguly
AU  - Edward F. Chang
PY  - 2026
DA  - 2026/09/14
JO  - Nature Neuroscience
DO  - 10.1038/s41593-026-02446-2
UR  - https://doi.org/10.1038/s41593-026-02446-2
AB  - Stroke and neurodegenerative diseases can impair speech and nonverbal gestures, limiting natural communication. Brain-computer interfaces (BCIs) aim to restore these functions by translating neural activity into commands for external devices, although prior work has primarily focused on decoding speech or gestures in isolation. Here we show that neural signals recorded with a single high-density electrocorticography implant can support simultaneous decoding of speech and gestures in people with paralysis. We first show that isolated upper-limb and orofacial movements can be reliably decoded among three participants. Using parallel speech and gesture decoders, we then enabled participants to control a personalized virtual avatar by attempting speech and gestures simultaneously or in isolation. Training models on both isolated and simultaneous data improved performance across behavioral contexts. These findings demonstrate that one cortical implant can support multi-effector control and provide a step toward BCIs that enable more natural communication for people with paralysis.
SN  - 1097-6256
SN  - 1546-1726
ER  -