Feeling the music: Audiotactile encoding of temporal structure in the human brain

Abstract

In everyday situations, like a rock party or an organ concert, we feel music vibrating through our bodies. How do these vibrotactile signals influence music processing? How do they aid auditory scene analysis? Combining psychophysics, fMRI and time-resolved EEG decoding, this work reveals how the brain encodes the temporal structure of music (beat and envelope) across audition and touch and uses this information to guide multisensory integration and segregation in simple and more complex perceptual scenes. Participants experienced monophonic and polyphonic piano pieces through auditory, vibrotactile and audiotactile stimulation. Vibrotactile signals improved the detection of a brief target embedded in music, establishing the functional relevance of audiotactile integration in naturalistic settings. fMRI and EEG multivariate decoding revealed that auditory and tactile beat information converged in planum temporale and parietal operculum, albeit through distinct neural dynamics and representations. Superior temporal cortices reliably encoded envelope information from audition, but only weakly from touch. Nevertheless, vibrotactile signals significantly enhanced neural encoding of auditory beat as early as 100 ms, and envelope representations from 250 ms onward. These encoding benefits were associated with superadditive interactions in primary auditory cortex, where tactile signals sharpen and amplify auditory envelope representations. In complex polyphonic music, touch further amplified the segregation and encoding of temporally coherent auditory streams. Our findings highlight the important, yet largely unexplored influence of touch on auditory processing, enriching music perception and supporting auditory scene analysis in real-world environments.