Neural correlates of probabilistic information processing in gesture-speech integration

Abstract

Semantic representation emerges from distributed multisensory modalities, yet a comprehensive understanding of the functional changing pattern within convergence zones or hubs integrating multisensory semantic information remains elusive. In this study, employing information-theoretic metrics, we quantified gesture and speech information, alongside their interaction, utilizing entropy and mutual information (MI). Neural implementation of these information-theoretic representations was characterized across three complementary experiments using high-definition transcranial direct current stimulation (HD-tDCS), functional magnetic resonance imaging (fMRI), and electroencephalography (EEG). HD-tDCS demonstrated a causal contribution of the left inferior frontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) to neural representations associated with gesture-speech MI. fMRI revealed that MI is represented across IFG and pMTG through distinct mechanisms: IFG preserves the relational structure predicted by MI, whereas pMTG supports discrimination of different levels of MI. EEG further showed that entropy primarily reflects transient modality-specific dynamics, whereas MI emerges during the N400-associated integration period. Cross-modal EEG-fMRI analyses demonstrated convergent MI-related neural representations with complementary temporal and spatial signatures. Together, these findings reveal a progressive organization of multisensory semantic integration, in which neural representations systematically track stimulus-level information-theoretic structure, with modality-specific uncertainty and cross-modal shared information expressed at different neural scales. This work establishes an information-theoretic framework that links the statistical structure of sensory information with its neural organization, providing new insights into how distributed inputs are coordinated to form coherent semantic representations.