Biology subjects

Lalor, E. C.

Publications and source records attributed to Lalor, E. C..

At least 19 recordsLinked to original sources

bioRxiv2026

Attention or prediction? Characterizing the top-down influence of predictive context on speech encoding

Theories of predictive coding propose that perception is the process of inferring the causes of our sensory input by comparing that input with predictions derived from our internal models of the world. Such predictive processes are thought to play a central role in language comprehension, however, robust neurophysiological evidence for such processes, particularly during natural speech perception, remains limited. Previous work has suggested that the early auditory encoding of words in natural speech is influenced by their preceding linguistic context. However, it remains unclear whether this effect is driven by prediction per se or dynamic modulations of attention based on contextual uncertainty. To distinguish between these alternatives, we recorded electroencephalography from 17 healthy adults while they listened to slightly changed audiobook. Specifically, we identified and replaced several unsurprising content words with more surprising words. We quantified the early auditory encoding of words using speech-envelope reconstruction accuracy within 100-ms time window after word onset and examined its relationship to word surprisal and contextual uncertainty. We found that more surprising words showed enhanced early auditory encoding despite matched contextual constraint. Moreover, the temporal profile of this enhancement depended on when the incoming speech signal diverged from the predicted phonological sequence, consistent with the emergence of prediction-error responses. Linear mixed-effects modeling further revealed that word surprisal had a substantially stronger influence on early auditory encoding than contextual uncertainty. Together, these findings indicate that the early auditory encoding of words during naturalistic speech perception is more strongly associated with predictive computations than with uncertainty-driven attentional gain. Significance StatementDuring natural speech comprehension, contextual information influences how the brain processes incoming sensory input. However, whether this context-dependent modulation of the early auditory encoding of words reflects predictive computations or dynamic changes in attentional gain has remained unresolved. By combining a naturalistic speech paradigm with a stimulus manipulation that varies word surprisal while controlling contextual uncertainty, we show that the early auditory encoding of words is driven by word surprisal under matched contextual constraint. Moreover, the temporal dynamics of this modulation closely follow the point at which the incoming speech signal departs from the predicted phonological sequence. These findings provide neurophysiological evidence that predictive computations contribute to the context- dependent modulation of early auditory processing during natural speech comprehension.

neuroscience↗

bioRxiv2026

Is that clear? Robust electrophysiological measures of the effects of prior knowledge on degraded speech perception.

Perception and sensation are not synonymous. Rather, perception is a process whereby sensory input is organized and interpreted in a behaviorally relevant way based on memory, experience, and context. One specific framework that is commonly invoked to explain perception is that of Bayesian inference. This framework casts perception as a probabilistic process whereby imprecise sensory data are combined with prior knowledge (prior) to determine what is consciously perceived (the posterior probability), which reflects the brains best guess as to the cause(s) of the sensory data. A striking behavioral example of how prior information can influence perception is seen in studies in which degraded speech is rendered intelligible by presenting information about the speech content in advance. Neurophysiological studies of this phenomenon have primarily focused on how it affects neural indices of low-level sensory encoding. The size of any reported effects on these indices tends to be much smaller - and much less consistent - than the notably large effects on perception that come with prior information. In the present study, we recorded EEG from 27 healthy adult participants (16 female) as they listened to degraded speech clips that were preceded by matching or mismatching text. Prior knowledge in the form of matching text led to a large perceptual pop-out effect when listening to degraded speech. Analyses of the resulting EEG revealed: 1) significant but relatively weak effects of prior information on EEG measures of the linguistic encoding of speech; and 2) a very large effect of prior information on an EEG signal that resembles a well-established neural index of perceptual evidence accumulation and that was strongly related to speech intelligibility ratings across participants. These EEG signals likely relate to separate components of a Bayesian inferential process during the predictive perception of degraded speech. As such, they have implications for understanding predictive perception more broadly and for future research on perceptual disturbances in clinical populations.

neuroscience↗

bioRxiv2026

Articulatory timing and form support distinct neural benefits during audiovisual speech

In noisy environments, visible speech articulations improve listening comprehension. The benefit derives from several sources, including articulatory timing and shape. Recent research has shown that visual cortex encodes a categorical representation of articulatory features and that visual speech can benefit both acoustic and phonetic feature processing separately. The present study advances the hypothesis that the shape of the articulators specifically influences the categorization of auditory speech in terms of its phonetic features. We tested this by linearly modeling electroencephalographic responses to natural, continuous speech (in noise) in terms of the acoustic and articulatory features of the speech. We compared the performance of these models in conditions where the speech was accompanied by a natural video of the speaker with their mouth visible, and a video where their mouth was covered by a dynamic ellipse obscuring articulatory shape but preserving dynamics. The dynamic mask reduced comprehension, neural processing of phonetic features, the associated multisensory benefits, and indices of visual-only linguistic processing over occipital scalp. Our findings support substantial visual involvement in speech comprehension, derived largely from the shape of the articulators. They also corroborate several proposals involving audiovisual speech processing hierarchy and the nature of the information contained in visible speech. HighlightsO_LIVisual speech provides at least two forms of information to enhance acoustic speech processing: redundant temporal dynamics and complementary articulatory information C_LIO_LICovering the mouth with a dynamic mask preserves horizontal and vertical lip movement information, but largely removes articulatory detail C_LIO_LIVisual speech with a mask preserves some general multisensory benefits but removes visual linguistic information and its ability to enhance auditory processing at the level of phonetic features. C_LI

neuroscience↗

bioRxiv2025

Spatial flexibility of distractor suppression in the tactile modality

Object sensing and manipulation rely on the somatosensory systems ability to relay relevant information about a skin location encoding tactile information about the object while suppressing irrelevant or distracting inputs on the skin--a process known as distractor suppression. Although tactile selective attention is flexible, capable of being allocated to non-contiguous skin areas on the hand and with spatial resolution limited to within a finger, the spatial properties of distractor suppression in touch remain poorly understood. In particular, studies have not explored the effects of proactively cueing distractor locations in touch, leaving open questions about whether active deployment of distractor suppression operates independently of attentional enhancement. To address this gap in knowledge, in this study, participants performed an amplitude discrimination task on the hand in the presence of distractors that were cued in advance. Our data revealed that validly cueing distractor locations improves discrimination accuracy and reduces reaction times of behaviorally-relevant tactile stimuli. The data also showed that behavior is unaffected when attended stimuli are flanked by distractors within or across fingers, suggesting that distractor suppression mechanisms can be split across different skin locations on the hand. These findings provide novel insights into the spatial allocation and flexibility of distractor suppression mechanisms in touch, and underscore the importance of proactive cueing in shaping these mechanisms.

neuroscience↗

bioRxiv2024

Chimeric music reveals an interaction of pitch and time in electrophysiological signatures of music encoding

Pitch and time are the essential dimensions defining musical melody. Recent electrophysiological studies have explored the neural encoding of musical pitch and time by leveraging probabilistic models of their sequences, but few have studied how the features might interact. This study examines these interactions by introducing "chimeric music," which pairs two distinct melodies, and exchanges their pitch contours and note onset-times to create two new melodies, thereby distorting musical pattern while maintaining the marginal statistics of the original pieces pitch and temporal sequences. Through this manipulation, we aimed to dissect the music processing and the interaction between pitch and time. Employing the temporal response function (TRF) framework, we analyzed the neural encoding of melodic expectation and musical downbeats in participants with varying levels of musical training. Our findings revealed differences in the encoding of melodic expectation between original and chimeric stimuli in both dimensions, with a significant impact of musical experience. This suggests that the structural violation due to decoupling the pitch and temporal structure affect expectation processing. In our analysis of downbeat encoding, we found an enhanced neural response when participants heard a note that aligned with the downbeat during music listening. In chimeric music, responses to downbeats were larger when the note was also a downbeat in the original music that provided the pitch sequence, indicating an effect of pitch structure on beat perception. This study advances our understanding of the neural underpinnings of music, emphasizing the significance of pitch-time interaction in the neural encoding of music. Significance StatementListening to music is a complex and multidimensional auditory experience. Recent studies have investigated the neural encoding of pitch and timing sequences in musical structure, but they have been studied independently. This study addresses the gap in understanding of how the interaction between pitch and time affects their encoding. By introducing "chimeric music," which decouples these dimensions in melodies, we investigate how this interaction influences the neural activities using EEG. Leveraging and the temporal response function (TRF) framework, we found that structural violations in pitch-time interactions impact musical expectation processing and beat perception. These results advance our knowledge of how the brain processes complex auditory stimuli like music, underscoring the critical role of pitch and time interactions in music perception.

neuroscience↗

bioRxiv2024

Dynamic modeling of EEG responses to natural speech reveals earlier processing of predictable words

In recent years, it has become clear that EEG indexes the comprehension of natural, narrative speech. One particularly compelling demonstration of this fact can be seen by regressing EEG responses to speech against measures of how individual words in that speech linguistically relate to their preceding context. This approach produces a so-called temporal response function that displays a centro-parietal negativity reminiscent of the classic N400 component of the event-related potential. One shortcoming of previous implementations of this approach is that they have typically assumed a linear, time-invariant relationship between the linguistic speech features and the EEG responses. In other words, the analysis typically assumes that the response has the same shape and timing for every word - and only varies (linearly) in terms of its amplitude. In the present work, we relax this assumption under the hypothesis that responses to individual words may be processed more rapidly when they are predictable. Specifically, we introduce a framework wherein the standard linear temporal response function can be modulated in terms of its amplitude, latency, and temporal scale based on the expectedness of the current and prior words. We use the proposed approach to model EEG recorded from a set of participants who listened to an audiobook narrated by a single talker, and a separate set of participants who attended to one of two concurrently presented audiobooks. We show that expected words are processed faster - evoking lower amplitude N400-like responses with earlier peaks - and that this effect is driven both by the words own predictability and the predictability of the immediately preceding word. Additional analysis suggests that this finding is not simply explained based on how quickly words can be disambiguated from their phonetic neighbors. As such, our study demonstrates that the timing and amplitude of brain responses to words in natural speech depend on their predictability. By accounting for these effects, our framework also improves the accuracy with which neural responses to natural speech can be modeled.

neuroscience↗

bioRxiv2024

Appropriate data segmentation improves speech encoding models

In recent decades, research on the neural processing of speech and language increasingly investigated ongoing responses to continuously presented naturalistic speech, allowing researchers to ask interesting questions about different representations of speech and their relationships. This requires statistical models that can dissect different sources of variance occurring in the processing of naturalistic speech. One commonly used family of models are temporal response functions (TRFs) which can predict neural responses to speech as a weighted combination of different features and points in time. TRFs model the brain as a linear time-invariant (LTI) system whose responses can be characterized by constant transfer functions. This implicitly assumes that the underlying signals are stationary, varying to a fixed degree around a constant mean. However, continuous neural recordings commonly violate this assumption. Here, we use simulations and EEG recordings to investigate how non-stationarities affect TRF models for continuous speech processing. Our results suggest that non-stationarities may impair the performance of TRF models, but that this can be partially remedied by dividing the data into shorter segments that approximate stationarity.

neuroscience↗

bioRxiv2023

Context and Attention Shape Electrophysiological Correlates of Speech-to-Language Transformation

To transform speech into words, the human brain must accommodate variability across utterances in intonation, speech rate, volume, accents and so on. A promising approach to explaining this process has been to model electroencephalogram (EEG) recordings of brain responses to speech. Contemporary models typically invoke speech categories (e.g. phonemes) as an intermediary representational stage between sounds and words. However, such categorical models are typically hand-crafted and therefore incomplete because they cannot speak to the neural computations that putatively underpin categorization. By providing end-to-end accounts of speech-to-language transformation, new deep-learning systems could enable more complete brain models. We here model EEG recordings of audiobook comprehension with the deep-learning system Whisper. We find that (1) Whisper provides an accurate, self-contained EEG model of speech-to-language transformation; (2) EEG modeling is more accurate when including prior speech context, which pure categorical models do not support; (3) EEG signatures of speech-to-language transformation depend on listener-attention.

neuroscience↗

bioRxiv2023

Electrophysiological indices of hierarchical speech processing differentially reflect the comprehension of speech in noise

The past few years have seen an increase in the use of encoding models to explain neural responses to natural speech. The goal of these models is to characterize how the human brain converts acoustic energy into distinct linguistic representations that enable everyday speech comprehension. For example, researchers have shown that electroencephalography (EEG) data can be modeled in terms of acoustic features of speech, such as its amplitude envelope or spectrogram, linguistic features such as phonemes and phoneme probability, and higher-level linguistic features like context-based word predictability. However, it is unclear how reliably EEG indices of these speech feature representations reflect comprehension in different listening conditions. To address this, we recorded EEG from neurotypical adults who listened to segments of an audiobook in various levels of background noise. We modeled how their EEG responses reflected a range of acoustic and linguistic speech features and how this tracking varied with behavior across noise levels. EEG tracking of nearly all examined features showed SNR-dependent changes in unique variance explained, with the largest changes occurring for linguistic features. We hypothesized that only higher-level feature tracking would predict behavior but instead found that both high and low-level features were associated with behavioral scores depending on the noise level. EEG markers of the influence of top-down, context-based prediction on bottom-up acoustic processing also correlated with behavior. These findings help characterize the relationship between brain and behavior by comprehensively linking hierarchical indices of neural speech processing to language comprehension metrics. SIGNIFICANCE STATEMENTAcoustic and linguistic features of speech have been shown to be consistently tracked by neural activity even in noisy conditions. However, it is unclear how signatures of low- and high-level features covary with one another and relate to behavior across these listening conditions. Here, we find that linguistic (phonetic feature and word probability-based feature) processing is affected by noise more than low-level acoustic feature processing. We also find that behavioral performance is associated with acoustic, phonetic, and lexical surprisal tracking, and that these associations depend on background noise levels. These results extend our understanding of how various speech features are comparatively reflected in electrical brain activity and how they relate to perception in challenging listening conditions.

neuroscience↗

bioRxiv2023

Incorporating models of subcortical processing improves the ability to predict EEG responses to natural speech

The goal of describing how the human brain responds to complex acoustic stimuli has driven auditory neuroscience research for decades. Often, a systems-based approach has been taken, in which neurophysiological responses are modeled based on features of the presented stimulus. This includes a wealth of work modeling electroencephalogram (EEG) responses to complex acoustic stimuli such as speech. Examples of the acoustic features used in such modeling include the amplitude envelope and spectrogram of speech. These models implicitly assume a direct mapping from stimulus representation to cortical activity. However, in reality, the representation of sound is transformed as it passes through early stages of the auditory pathway, such that inputs to the cortex are fundamentally different from the raw audio signal that was presented. Thus, it could be valuable to account for the transformations taking place in lower-order auditory areas, such as the auditory nerve, cochlear nucleus, and inferior colliculus (IC) when predicting cortical responses to complex sounds. Specifically, because IC responses are more similar to cortical inputs than acoustic features derived directly from the audio signal, we hypothesized that linear mappings (temporal response functions; TRFs) fit to the outputs of an IC model would better predict EEG responses to speech stimuli. To this end, we modeled responses to the acoustic stimuli as they passed through the auditory nerve, cochlear nucleus, and inferior colliculus before fitting a TRF to the output of the modeled IC responses. Results showed that using model-IC responses in traditional systems analyses resulted in better predictions of EEG activity than using the envelope or spectrogram of a speech stimulus. Further, it was revealed that model-IC derived TRFs predict different aspects of the EEG than acoustic-feature TRFs, and combining both types of TRF models provides a more accurate prediction of the EEG response.x

neuroscience↗

bioRxiv2021

The integration of continuous audio and visual speech in a cocktail-party environment depends on attention

In noisy, complex environments, our ability to understand audio speech benefits greatly from seeing the speakers face. This is attributed to the brains ability to integrate audio and visual information, a process known as multisensory integration. In addition, selective attention to speech in complex environments plays an enormous role in what we understand, the so-called cocktail-party phenomenon. But how attention and multisensory integration interact remains incompletely understood. While considerable progress has been made on this issue using simple, and often illusory (e.g., McGurk) stimuli, relatively little is known about how attention and multisensory integration interact in the case of natural, continuous speech. Here, we addressed this issue by analyzing EEG data recorded from subjects who undertook a multisensory cocktail-party attention task using natural speech. To assess multisensory integration, we modeled the EEG responses to the speech in two ways. The first assumed that audiovisual speech processing is simply a linear combination of audio speech processing and visual speech processing (i.e., an A+V model), while the second allows for the possibility of audiovisual interactions (i.e., an AV model). Applying these models to the data revealed that EEG responses to attended audiovisual speech were better explained by an AV model than an A+V model, providing evidence for multisensory integration. In contrast, unattended audiovisual speech responses were best captured using an A+V model, suggesting that multisensory integration is suppressed for unattended speech. Follow up analyses revealed some limited evidence for early multisensory integration of unattended AV speech, with no integration occurring at later levels of processing. We take these findings as evidence that the integration of natural audio and visual speech occurs at multiple levels of processing in the brain, each of which can be differentially affected by attention.

neuroscience↗

bioRxiv2021

A linguistic representation in the visual system underlies successful lipreading

There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Surprisingly, much of this ambiguity stems from a lack of behaviorally grounded neurophysiological findings. To address this, we conducted an experiment in which human observers rehearsed audiovisual speech for the purpose of lipreading silent versions during testing. Using a combination of computational modeling, electroencephalography, and simultaneously recorded behavior, we show that the visual system produces its own specialized representation of speech that is 1) well-described by categorical linguistic units ("visemes") 2) dissociable from lip movements, and 3) predictive of lipreading ability. These findings contradict a long-held view that visual speech processing co-opts auditory cortex after early visual processing stages. Consistent with hierarchical accounts of visual and audiovisual speech perception, our findings show that visual cortex performs at least a basic level of linguistic processing.

neuroscience↗

bioRxiv2021

Envelope reconstruction of speech and music highlights unique tracking of speech at low frequencies

The human brain tracks amplitude fluctuations of both speech and music, which reflects acoustic processing in addition to the processing of higher-order features and ones cognitive state. Comparing neural tracking of speech and music envelopes can elucidate stimulus-general mechanisms, but direct comparisons are confounded by differences in their envelope spectra. Here, we use a novel method of frequency-constrained reconstruction of stimulus envelopes using EEG recorded during passive listening. We expected to see music reconstruction match speech in a narrow range of frequencies, but instead we found that speech was reconstructed better than music for all frequencies we examined. Additionally, speech envelope tracking at low frequencies, below 1 Hz, was uniquely associated with increased weighting over parietal channels. Our results highlight the importance of low-frequency speech tracking and its origin from speech-specific processing in the brain.

neuroscience↗

bioRxiv2020

More than Words: Neurophysiological Correlates of Semantic Dissimilarity Depend on Comprehension of the Speech Narrative

Speech comprehension relies on the ability to understand the meaning of words within a coherent context. Recent studies have attempted to obtain electrophysiological indices of this process by modelling how brain activity is affected by a words semantic dissimilarity to preceding words. While the resulting indices appear robust and are strongly modulated by attention, it remains possible that, rather than capturing the contextual understanding of words, they may actually reflect word-to-word changes in semantic content without the need for a narrative-level understanding on the part of the listener. To test this possibility, we recorded EEG from subjects who listened to speech presented in either its original, narrative form, or after scrambling the word order by varying amounts. This manipulation affected the ability of subjects to comprehend the narrative content of the speech, but not the ability to recognize the individual words. Neural indices of semantic understanding and low-level acoustic processing were derived for each scrambling condition using the temporal response function (TRF) approach. Signatures of semantic processing were observed for conditions where speech was unscrambled or minimally scrambled and subjects were able to understand the speech. The same markers were absent for higher levels of scrambling when speech comprehension dropped below chance. In contrast, word recognition remained high and neural measures related to envelope tracking did not vary significantly across the different scrambling conditions. This supports the previous claim that electrophysiological indices based on the semantic dissimilarity of words to their context reflect a listeners understanding of those words relative to that context. It also highlights the relative insensitivity of neural measures of low-level speech processing to speech comprehension.

neuroscience↗

bioRxiv2020

Co-existence of prediction and error signals in electrophysiological responses to natural speech

Prior knowledge facilitates perception and allows us to interpret our sensory environment. However, the neural mechanisms underlying this process remain unclear. Theories of predictive coding propose that feedback connections between cortical levels carry predictions about upcoming sensory events whereas feedforward connections carry the error between the prediction and the sensory input. Although predictive coding has gained much ground as a viable mechanism for perception, in the context spoken language comprehension it lacks empirical support using more naturalistic stimuli. In this study, we investigated theories of predictive coding using continuous, everyday speech. EEG recordings from human participants listening to an audiobook were analysed using a 2-stage regression framework. This tested the effect of top-down linguistic information, estimated using computational language models, on the bottom-up encoding of acoustic and phonetic speech features. Our results show enhanced encoding of both semantic predictions and surprising words, based on preceding context. This suggests that signals pertaining to prediction and error units can be observed in the same electrophysiological responses to natural speech. In addition, temporal analysis of these signals reveals support for theories of predictive coding that propose that perception is first biased towards what is expected followed by what is informative. Significance StatementOver the past two decades, predictive coding has grown in popularity as an explanatory mechanism for perception. However, there has been lack of empirical support for this theory in research studying natural speech comprehension. We address this issue by developing an analysis framework that tests the effects of top-down linguistic information on the auditory encoding of continuous speech. Our results provide evidence for the co-existence of prediction and error signals and support theories of predictive coding using more naturalistic stimuli.

neuroscience↗

bioRxiv2020

General auditory and speech-specific contributions to cortical envelope tracking revealed using auditory chimeras

1.In recent years research on natural speech processing has benefited from recognizing that low frequency cortical activity tracks the amplitude envelope of natural speech. However, it remains unclear to what extent this tracking reflects speech-specific processing beyond the analysis of the stimulus acoustics. In the present study, we aimed to disentangle contributions to cortical envelope tracking that reflect general acoustic processing from those that are functionally related to processing speech. To do so, we recorded EEG from subjects as they listened to "auditory chimeras" - stimuli comprised of the temporal fine structure (TFS) of one speech stimulus modulated by the amplitude envelope (ENV) of another speech stimulus. By varying the number of frequency bands used in making the chimeras, we obtained some control over which speech stimulus was recognized by the listener. No matter which stimulus was recognized, envelope tracking was always strongest for the ENV stimulus, indicating a dominant contribution from acoustic processing. However, there was also a positive relationship between intelligibility and the tracking of the perceived speech, indicating a contribution from speech specific processing. These findings were supported by a follow-up analysis that assessed envelope tracking as a function of the (estimated) output of the cochlea rather than the original stimuli used in creating the chimeras. Finally, we sought to isolate the speech-specific contribution to envelope tracking using forward encoding models and found that indices of phonetic feature processing tracked reliably with intelligibility. Together these results show that cortical speech tracking is dominated by acoustic processing, but also reflects speech-specific processing. This work was supported by a Career Development Award from Science Foundation Ireland (CDA/15/3316) and a grant from the National Institute on Deafness and Other Communication Disorders (DC016297). The authors thank Dr. Aaron Nidiffer, Dr. Aisling OSullivan, Thomas Stoll and Lauren Szymula for assistance with data collection, and Dr. Nathaniel Zuk, Dr. Aaron Nidiffer, Dr. Aisling OSullivan for helpful comments on this manuscript. 2. Significance StatementActivity in auditory cortex is known to dynamically track the energy fluctuations, or amplitude envelope, of speech. Measures of this tracking are now widely used in research on hearing and language and have had a substantial influence on theories of how auditory cortex parses and processes speech. But, how much of this speech tracking is actually driven by speech-specific processing rather than general acoustic processing is unclear, limiting its interpretability and its usefulness. Here, by merging two speech stimuli together to form so-called auditory chimeras, we show that EEG tracking of the speech envelope is dominated by acoustic processing, but also reflects linguistic analysis. This has important implications for theories of cortical speech tracking and for using measures of that tracking in applied research.

neuroscience↗

bioRxiv2020

Attention differentially affects acoustic and phonetic feature encoding in a multispeaker environment.

Humans have the remarkable ability to selectively focus on a single talker in the midst of other competing talkers. The neural mechanisms that underlie this phenomenon remain incompletely understood. In particular, there has been longstanding debate over whether attention operates at an early or late stage in the speech processing hierarchy. One way to better understand this is to examine how attention might differentially affect neurophysiological indices of hierarchical acoustic and linguistic speech representations. In this study, we do this by using encoding models to identify neural correlates of speech processing at various levels of representation. Specifically, using EEG recorded during a "cocktail party" attention experiment, we show that phonetic feature processing is evident for attended, but not unattended speech. Furthermore, we show that attention specifically enhances isolated indices of phonetic feature processing, but that such attention effects are not apparent for isolated measures of acoustic processing. These results provide new insights into the effects of attention on different pre-lexical representations of speech, insights that complement recent anatomical accounts of the hierarchical encoding of attended speech. Furthermore, our findings support the notion that - for attended speech - phonetic features are processed as a distinct stage, separate from the processing of the speech acoustics.

neuroscience↗

bioRxiv2020

Neurophysiological indices of audiovisual speech integration are enhanced at the phonetic level for speech in noise

Seeing a speakers face benefits speech comprehension, especially in challenging listening conditions. This perceptual benefit is thought to stem from the neural integration of visual and auditory speech at multiple stages of processing, whereby movement of a speakers face provides temporal cues to auditory cortex, and articulatory information from the speakers mouth can aid recognizing specific linguistic units (e.g., phonemes, syllables). However it remains unclear how the integration of these cues varies as a function of listening conditions. Here we sought to provide insight on these questions by examining EEG responses to natural audiovisual, audio, and visual speech in quiet and in noise. Specifically, we represented our speech stimuli in terms of their spectrograms and their phonetic features, and then quantified the strength of the encoding of those features in the EEG using canonical correlation analysis. The encoding of both spectrotemporal and phonetic features was shown to be more robust in audiovisual speech responses then what would have been expected from the summation of the audio and visual speech responses, consistent with the literature on multisensory integration. Furthermore, the strength of this multisensory enhancement was more pronounced at the level of phonetic processing for speech in noise relative to speech in quiet, indicating that listeners rely more on articulatory details from visual speech in challenging listening conditions. These findings support the notion that the integration of audio and visual speech is a flexible, multistage process that adapts to optimize comprehension based on the current listening conditions. Significance StatementDuring conversation, visual cues impact our perception of speech. Integration of auditory and visual speech is thought to occur at multiple stages of speech processing and vary flexibly depending on the listening conditions. Here we examine audiovisual integration at two stages of speech processing using the speech spectrogram and a phonetic representation, and test how audiovisual integration adapts to degraded listening conditions. We find significant integration at both of these stages regardless of listening conditions, and when the speech is noisy, we find enhanced integration at the phonetic stage of processing. These findings provide support for the multistage integration framework and demonstrate its flexibility in terms of a greater reliance on visual articulatory information in challenging listening conditions.

neuroscience↗