Hierarchical Uncertainty Transformation During Multimodal Comprehension
- 1Institute of Psychology, Chinese Academy of Sciences
Abstract
Human communication requires combining information from different modalities even when each source is incomplete or ambiguous. Yet how uncertainty is coordinated as distinct semantic representations interact and converge remains unclear. Here we propose a hierarchical uncertainty transformation framework and test it across four gesture-speech experiments combining behavioral measures, information-theoretic modelling, and electroencephalography. Rather than being uniformly reduced as information accumulates, uncertainty changed its computational form across three interdependent levels: uncertainty within modality-specific semantic representations, bidirectional but asymmetric changes during cross-modal interaction, and uncertainty associated with multisensory convergence. These computational distinctions were accompanied by differentiated but temporally overlapping neural associations. Modality-specific uncertainty was expressed relatively early, whereas cross-modal transformation and convergence were most prominently associated with activity in the N400 range. Critically, both the neural expression and statistical coupling of these computational quantities varied with the semantic information available from gesture and speech. Together, these findings show that multimodal comprehension involves a context-dependent reorganization of uncertainty across interacting representations, providing a computational framework for understanding how coherent meaning emerges from heterogeneous and uncertain information.
Related
- Electrophysiological Indices of Hierarchical Speech Processing Differentially Reflect the Comprehension of Speech in Noise
- Beta bursts carry a graded signal of semantic updating during naturalistic narrative comprehension
- What Shapes Brain Alignment of Large Language Models during Dialogue? Insights from Language Production and Comprehension
- A domain-general basal ganglia mechanism for inhibitory control during language comprehension
- Speech Tracking in fMRI Reveals a Distinct Response to Sentence Boundaries During Comprehension
BibTeX
@misc{zhao2026hierarchical,
title = {{Hierarchical Uncertainty Transformation During Multimodal Comprehension}},
author = {W. Zhao and J. Yang and Z. Li and Y. Du},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.29.762725},
url = {https://www.biorxiv.org/content/10.64898/2026.10.29.762725v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Hierarchical Uncertainty Transformation During Multimodal Comprehension
AU - W. Zhao
AU - J. Yang
AU - Z. Li
AU - Y. Du
PY - 2026
DA - 2026/10/29
PB - bioRxiv
DO - 10.64898/2026.10.29.762725
UR - https://www.biorxiv.org/content/10.64898/2026.10.29.762725v1
AB - Human communication requires combining information from different modalities even when each source is incomplete or ambiguous. Yet how uncertainty is coordinated as distinct semantic representations interact and converge remains unclear. Here we propose a hierarchical uncertainty transformation framework and test it across four gesture-speech experiments combining behavioral measures, information-theoretic modelling, and electroencephalography. Rather than being uniformly reduced as information accumulates, uncertainty changed its computational form across three interdependent levels: uncertainty within modality-specific semantic representations, bidirectional but asymmetric changes during cross-modal interaction, and uncertainty associated with multisensory convergence. These computational distinctions were accompanied by differentiated but temporally overlapping neural associations. Modality-specific uncertainty was expressed relatively early, whereas cross-modal transformation and convergence were most prominently associated with activity in the N400 range. Critically, both the neural expression and statistical coupling of these computational quantities varied with the semantic information available from gesture and speech. Together, these findings show that multimodal comprehension involves a context-dependent reorganization of uncertainty across interacting representations, providing a computational framework for understanding how coherent meaning emerges from heterogeneous and uncertain information.
ER -