bioRxiv Science⌕ Search

Biology subjects

Vanthornhout, J.

Publications and source records attributed to Vanthornhout, J..

At least 19 recordsLinked to original sources
bioRxiv2026

Interpersonal Synchronization of Brain and Body Tracks Attention and Listening Engagement

Purpose: Speech perception is a fundamental part of everyday communication that relies on more than simple identification of words and sentences. Attention and listening engagement both contribute to speech perception, while representing distinct aspects of the listening experience. Attention is typically associated with cognitive focus, whereas listening engagement additionally involves cognitive and affective immersion in sound. Despite their importance, these states remain difficult to disentangle, behaviorally and physiologically. Both have been linked to interpersonal synchronization (the synchronization of biobehavioral signals across individuals), raising questions about what this synchronization actually reflects. Method: In this study, we disentangled attention and listening engagement by independently manipulating both factors within a single experiment. Thirty participants listened to two simultaneously presented streams of meaningful speech and were instructed to focus on only one. Both attended and unattended stimuli were designed to be either engaging or non-engaging. Neural activity was recorded using EEG, while physiological responses were measured using heart rate and electrodermal activity. Results: Interpersonal synchronization was computed from neural and bodily signals, alongside a self-report measure of listening engagement and auditory attention decoding (AAD), a neural measure of selective attention. Interpersonal synchronization of all three modalities significantly predicted listening engagement, whereas neural interpersonal synchronization was the only measure that significantly predicted attention. These findings suggest that attention is primarily driven by cognitive processes represented in the brain, while listening engagement additionally involves affective processes that are more strongly reflected in bodily responses. Conclusions: Overall, this study demonstrates that different forms of interpersonal synchronization reflect distinct dimensions of the listening experience and supports interpersonal synchronization as a potential objective marker of listening engagement.

neuroscience↗
bioRxiv2025

Neural Tracking of Audiovisual Effects in Noise Using Deep Neural Network-Generated Virtual Humans

This study investigates the effectiveness of Deep Neural Network (DNN)-generated virtual humans in enhancing audiovisual speech perception in noisy environments, with a focus on using neural measures to quantify these effects. Lip movements are essential for speech comprehension, especially when auditory cues are degraded by noise. Traditional recording methods produce high-quality audiovisual materials but are resource intensive. This research explores the use of DNN avatars as a promising alternative, utilizing a commercially available tool to create realistic virtual humans. The study included both simple sentences and a short story to improve ecological validity. Eleven young, normal-hearing participants proficient in Flemish-Dutch listened to semantically meaningful sentences and a short story with various speaker types: a female FACS avatar, male and female DNN avatars, and a video of a human male speaker. The study included behavioral measures which consisted of an adaptive recall procedure and an adaptive rate procedure and electrophysiological measures consisting of neural tracking. Findings in the adaptive recall procedure showed consistent audiovisual benefits, with the human speaker offering the greatest benefit (-4.75 dB SNR), followed by the DNN avatar (-4.00 dB SNR) and the FACS avatar (-1.55 dB SNR). Additionally in the adaptive rate procedure, the DNN avatar improved speech intelligibility, with average SRTs enhancing from -7.17 dB SNR (audio-only) to -9.02 dB SNR (audiovisual). The results from the neural tracking procedure indicated that most participants experienced audiovisual benefits, particularly in the -9 dB SNR range, revealing that audiovisual cues provided by DNN avatars can enhance speech perception, validating these avatars as effective tools for studying audiovisual effects when using both behavioral measures and electrophysiological measures.

neuroscience↗
bioRxiv2025

Comparison of Neural Tracking and Spectral Entropy in Patients with Disorders of Consciousness

ObjectivesThis study aims to explore the brain responses of patients with disorders of consciousness (DoC) to natural speech. Specifically, it focuses on a key characteristic of natural speech: the speech envelope. To achieve this, we employed two distinct measures. The first evaluates how effectively the patients brain activity "tracks" the speech envelope, called neural tracking. The second assesses the complexity of the brains responses to the speech stimulus, measured through spectral entropy. These two measures are then compared in their association with the patients clinical diagnosis and their level of behavioral responsiveness, both of which were assessed using the Coma Recovery Scale-Revised (CRS-R). DesignFour patients with DoC participated in this study, during which their brain activity was recorded using electroen-cephalography (EEG). At the same time, they listened to a narrated story in both Dutch and Swedish. Additionally, EEG baseline recordings were collected. We employed a backward modeling approach to evaluate the speech envelopes neural tracking. This technique involves training a model to map the relationship between EEG signals and the corresponding speech envelope. Once the model is trained, it can use unseen EEG data to reconstruct the speech envelope, which is then compared to the original speech envelope to assess how effectively the patient processed the auditory stimulus. For the behavioral assessment, we recalculated the CRS-R score of each patient into the CSR-R index, a more meaningful score that utilizes all the information contained in the CRS-R instead of only the highest scores on each subscale. ResultsOur findings revealed positive correlations between spectral entropy and the CRS-R index, which were more pronounced during the listening conditions than the baseline. While neural tracking of the speech envelope did not correlate with the CRS-R index, it did exhibit a positive association with CRS-R diagnoses, indicating that patients with better clinical diagnoses demonstrated higher levels of neural tracking. Additionally, we identified an interaction effect between spectral entropy and neural tracking. Specifically, higher levels of neural tracking were associated with a stronger positive relationship between spectral entropy and the CRS-R index. In contrast, when neural tracking was lower, this relationship disappeared. ConclusionThis study demonstrated the potential of neural tracking and spectral entropy as complementary tools to investigate patients with DoC. Spectral entropy proved valuable for assessing behavioral responsiveness, while neural tracking shows promise in assessing the DoC diagnosis. Terms: disorders of consciousness (DoC), neural tracking, speech envelope, spectral entropy

neuroscience↗
bioRxiv2025

Large Language Models Reveal the Neural Tracking of Linguistic Context in Attended and Unattended Multi-Talker Speech

Large language models (LLMs) capture long-range contextual structure in natural language and have recently been shown to align with the human brains contextualized linguistic encoding. This makes them a promising computational probe for studying how context-dependent linguistic information is represented during natural speech perception. Speech perception often occurs in multi-talker environments, where attention must dynamically select among competing streams, yet how contextual information from attended and unattended speech is neurally encoded remains underexplored. Here, we investigate how auditory attention modulates neural tracking of context-dependent linguistic representations using electrocorticography (ECoG) and stereoelectroencephalography (sEEG) recordings from three epilepsy patients engaged in a two-conversation "cocktail party" paradigm. To model neural responses to attended and unattended speech streams, we used contextual word embeddings generated by large language models. We find that LLM-derived features reliably predict brain activity for the attended stream and that contextual information from the unattended stream also contributes to neural prediction. Importantly, these contributions extend beyond low-level acoustic features and shallow syntactic information, and depend on the surrounding linguistic context. Moreover, neural tracking of the unattended stream reflects shorter-range contextual integration than that of the attended stream. Together, these findings indicate that neural responses to speech reflect context-dependent linguistic representations from multiple concurrent speech streams, with attention modulating the depth and timescale of contextual integration. Our results highlight the utility of LLMs for probing higher-level linguistic representations in complex, naturalistic listening environments.

neuroscience↗
bioRxiv2025

Interpersonal synchronization as an objective measure of listening engagement

Listening engagement plays a crucial role in effective communication and knowledge transfer, reflecting a state of deep absorption in an auditory stimulus. The available behavioral assessments of listening engagement are limited in capturing its multidimensional nature. Measuring listening engagement objectively using brain and body signals is a compelling alternative. In particular, interpersonal synchronization--observed through synchronized neural and physiological responses among listeners--may offer a promising objective measure. In this study, we examined synchronization of brain activity, heart rate, and electrodermal activity between participants while they listened to engaging and non-engaging stories. Subjective engagement ratings were collected as a ground truth reference. Results showed higher interpersonal synchronization across all modalities when participants listened to engaging stories compared to non-engaging ones, with significant correlations between synchronization measures and ratings of subjective engagement. These results confirm that synchronization may be a reliable, objective marker of listening engagement. We anticipate that these results will provide a powerful framework for future research in relevant domains, including auditory neuroscience, education, and communication science, and enable impactful real-world applications, such as automatically tracking audience engagement in education, entertainment, and broader societal contexts.

neuroscience↗
bioRxiv2024

Neural tracking of the speech envelope predicts binaural unmasking

Binaural unmasking is the remarkable phenomenon that it is substantially easier to detect a signal in noise, when the interaural parameters of the signal are different from those of the noise - a mechanism that comes in handy in so-called cocktail party scenarios. In this study, we investigated the effect of binaural unmasking on neural tracking of the speech envelope. We measured EEG in 8 participants who listened to speech in noise at a fixed signal-to-noise ratio (-12 dB or -9 dB, depending on the speech material), in two conditions: one where speech and noise had the same interaural phase difference (both speech and noise having an opposite waveform across ears, S{pi}N{pi}), and one where the interaural phase difference of the speech was different from that of the noise (only the speech having an opposite waveform across ears, S{pi}N0). We measured a clear benefit of binaural unmasking in behavioral speech understanding scores, accompanied with increased neural tracking of the speech envelope. Moreover, analyzing the temporal response functions revealed that binaural unmasking also resulted in decreased peak latencies and increased peak amplitudes. Our results are consistent with previous research using auditory evoked potentials and steady-state responses to quantify binaural unmasking at cortical levels. Moreover, they confirm that neural tracking of speech is modulated by speech understanding, even if the acoustic signal-to-noise ratio is kept constant. Significance StatementBinaural unmasking in an important contributing factor to speech understanding in noisy environments. This is the first time that this benefit in speech understanding is measured objectively with EEG. From a clinical perspective, these results could enable the evalu-ation of binaural speech understanding mechanisms in populations for whom behavioral measures are difficult to obtain, such as young children or people with cognitive impair-ment. Moreover, behavioral research has shown that many pathologies (e.g., asymmetric hearing loss, auditory neuropathy, and age-related deficits) are more sensitive to tests that contain a binaural processing component. Our paradigm could enable the detection of such pathologies with an objective approach using neural tracking of speech.

neuroscience↗
bioRxiv2024

Classifying native versus foreign speech perception from EEG using linguistic speech features

When a person listens to natural speech, the relation between features of the speech signal and the corresponding evoked electroencephalogram (EEG) is indicative of neural processing of the speech signal. Using linguistic representations of speech, we investigate the differences in neural processing between speech in a native and foreign language that is not understood. We conducted experiments using three stimuli: a comprehensible language, an incomprehensible language, and randomly shuffled words from a comprehensible language, while recording the EEG signal of native Dutch-speaking participants. We modeled the neural tracking of linguistic features of the speech signals using a deep-learning model in a match-mismatch task that relates EEG signals to speech, while accounting for lexical segmentation features reflecting acoustic processing. The deep learning model effectively classifies languages. We also observed significant differences in tracking patterns between comprehensible and incomprehensible speech stimuli within the same language. It demonstrates the potential of deep learning frameworks in measuring speech understanding objectively.

neuroscience↗
bioRxiv2024

Neural substrates and behavioral relevance of speech envelope tracking: evidence from post-stroke aphasia

Neural tracking of the low-frequency temporal envelope of speech has emerged as a prominent tool to investigate the neural mechanisms of natural speech processing in the brain. However, there is ongoing debate regarding the functional role of neural envelope tracking. In this context, our study aims to offer a novel perspective by investigating the critical brain areas and behavioral skills required for neural envelope tracking in aphasia, a language disorder characterized by impaired neural envelope tracking. We analyzed an EEG dataset of 39 individuals with post-stroke aphasia suffering a left-hemispheric stroke who listened to natural speech. Our analysis involved lesion mapping, where left lesioned brain voxels served as binary features to predict neural envelope tracking measures. We also examined the behavioral correlates of receptive language, naming, and auditory processing (via rise time discrimination task) skills. The lesion mapping analysis revealed that lesions in language areas, such as the middle temporal gyrus, supramarginal gyrus and angular gyrus, were associated with poorer neural envelope tracking. Additionally, neural tracking was related to auditory processing skills and language (receptive and naming) skills. However, the effects on language skills were less robust, possibly due to ceiling effects in the language scores. Our [fi]ndings highlight the importance of central brain areas implicated in language understanding, extending beyond the primary auditory cortex, and emphasize the role of intact auditory processing and language abilities in effectively processing the temporal envelope of speech. Collectively, these [fi]ndings underscore the signi[fi]cance of neural envelope tracking beyond mere audibility and acoustic processes. Signi[fi]cance statementWhile some studies have proposed that neural envelope tracking primarily relates to audibility and acoustic speech processes, others have suggested its involvement in actual speech and language comprehension. By investigating the critical brain areas and behavioral skills essential in aphasia, we argue for a broader signi[fi]cance of neural envelope tracking in language processing. Furthermore, our [fi]ndings highlight a speci[fi]city among individuals with aphasia, indicating its correlation with lesions in temporal brain regions associated with receptive language functions. This addresses the signi[fi]cant heterogeneity in lesion characteristics present among individuals with aphasia and suggests the potential of neural tracking as an EEG-based tool for speci[fi]cally assessing receptive language abilities in this population.

neuroscience↗
bioRxiv2023

Auditory steady-state responses: multiplexed amplitude modulation frequencies to reduce recording time

ObjectivesThis study investigated the efficiency of a multiplexed amplitude-modulated (AM) stimulus in eliciting auditory steady-state responses (ASSR). The multiplexed AM stimulus was created by simultaneously modulating speech-shaped noise with three frequencies chosen to elicit different neural generators: 3.1, 40.1, and 102.1 Hz. For comparison, a single AM stimulus was created for each of these frequencies, resulting in three single AM conditions and one multiplex AM condition. DesignTwenty-two bilaterally normal-hearing participants (18 female) listened for 8 minutes to each type of stimuli. The analysis compared the signal-to-noise ratios (SNRs) and amplitudes of the evoked responses to the single and multiplexed conditions. ResultsThe results revealed that the SNRs elicited by single AM conditions were, on average, 1.61 dB higher than those evoked by the multiplexed AM condition (p < 0.05). The single conditions consistently produced a significantly higher SNR when examining various stimulus durations ranging from 1 to 8 minutes. Despite these SNR differences, the frequency spectrum was very similar across and within subjects. Additionally, the sensor space patterns across the scalp demonstrated similar trends between the single and multiplexed stimuli for both SNR and amplitudes. Both the single and multiplexed conditions evoked significant ASSRs within subjects. On average, the multiplexed AM stimulus took 31 minutes for the lower bound of the 95% prediction interval to cross the significance threshold across all three frequencies. In contrast, the single AM stimuli took 45 minutes and 42 seconds. ConclusionThese findings show that the multiplexed AM stimulus is a promising method to reduce the recording time when simultaneously obtaining information from various neural generators.

neuroscience↗
bioRxiv2023

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker's mouth is occluded

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

neuroscience↗
bioRxiv2023

Robust neural tracking of linguistic speech representations using a convolutional neural network.

ObjectiveWhen listening to continuous speech, populations of neurons in the brain track different features of the signal. Neural tracking can be measured by relating the electroencephalography (EEG) and the speech signal. Recent studies have shown a significant contribution of linguistic features over acoustic neural tracking using linear models. However, linear models cannot model the nonlinear dynamics of the brain. To overcome this, we use a convolutional neural network (CNN) that relates EEG to linguistic features using phoneme or word onsets as a control and has the capacity to model non-linear relations. ApproachWe integrate phoneme- and word-based linguistic features (phoneme surprisal, cohort entropy, word surprisal and word frequency) in our nonlinear CNN model and investigate if they carry additional information on top of lexical features (phoneme and word onsets). We then compare the performance of our nonlinear CNN with that of a linear encoder and a linearized CNN. Main resultsFor the non-linear CNN, we found a significant contribution of cohort entropy over phoneme onsets and of word surprisal and word frequency over word onsets. Moreover, the non-linear CNN outperformed the linear baselines. SignificanceMeasuring coding of linguistic features in the brain is important for auditory neuroscience research and applications that involve objectively measuring speech understanding. With linear models, this is measurable, but the effects are very small. The proposed non-linear CNN model yields larger differences between linguistic and lexical models and, therefore, could show effects that would otherwise be unmeasurable and may, in the future, lead to improved within-subject measures and shorter recordings. Index TermsEEG decoding, speech processing, CNN, linguistics.

neuroscience↗
bioRxiv2023

Exploring neural tracking of acoustic and linguistic speech representations in individuals with post-stroke aphasia

Aphasia is a communication disorder that affects processing of language at different levels (e.g., acoustic, phonological, semantic). Recording brain activity via EEG while people listen to a continuous story allows to analyze brain responses to acoustic and linguistic properties of speech. When the neural activity aligns with these speech properties, it is referred to as neural tracking. Even though measuring neural tracking of speech may present an interesting approach to studying aphasia in an ecologically valid way, it has not yet been investigated in individuals with stroke-induced aphasia. Here, we explored processing of acoustic and linguistic speech representations in individuals with aphasia in the chronic phase after stroke and age-matched healthy controls. We found decreased neural tracking of acoustic speech representations (envelope and envelope onsets) in individuals with aphasia. In addition, word surprisal displayed decreased amplitudes in individuals with aphasia around 195 ms over frontal electrodes, although this effect was not corrected for multiple comparisons. These results show that there is potential to capture language processing impairments in individuals with aphasia by measuring neural tracking of continuous speech. However, more research is needed to validate these results. Nonetheless, this exploratory study shows that neural tracking of naturalistic, continuous speech presents a powerful approach to studying aphasia. Key pointsO_LIIndividuals with aphasia display decreased encoding of acoustic speech properties (envelope and its onsets) in comparison to healthy controls. C_LIO_LINeural responses to word surprisal reveal decreased amplitudes in individuals with aphasia around 195 ms processing time (not corrected for multiple comparisons). C_LIO_LINeural tracking of natural speech can be used to study speech processing impairments in aphasia. C_LI

neuroscience↗
bioRxiv2022

Heard or understood? Neural tracking of language features in a comprehensible story, an incomprehensible story and a word list

Speech comprehension is a complex neural process on which relies on activation and integration of multiple brain regions. In the current study, we evaluated whether speech comprehension can be investigated by neural tracking. Neural tracking is the phenomenon in which the brain responses time-lock to the rhythm of specific features in continuous speech. These features can be acoustic, i.e., acoustic tracking, or derived from the content of the speech using language properties, i.e., language tracking. We evaluated whether neural tracking of speech differs between a comprehensible story, an incomprehensible story, and a word list. We evaluated the neural responses to speech of 19 participants (6 men). No significant difference regarding acoustic tracking was found. However, significant language tracking was only found for the comprehensible story. The most prominent effect was visible to word surprisal, a language feature at the word level. The neural response to word surprisal showed a prominent negativity between 300 and 400 ms, similar to the N400 in evoked response paradigms. This N400 was significantly more negative when the story was comprehended, i.e., when words could be integrated in the context of previous words. These results show that language tracking can capture the effect of speech comprehension. Significance statementMost neural tracking studies focus on how the brain tracks acoustic speech features. However, whether acoustic tracking reflects speech comprehension is unclear. Therefore, in the pursuit of finding a neural marker for speech comprehension, language tracking might be a more suitable candidate. The results of this study showed that, indeed, language tracking can capture the effect of speech comprehension. This indicates that language tracking is a potential neural marker of speech comprehension. Such a neural marker would allow testing speech comprehension in populations that are currently difficult to test with behavioral tests, such as young children and persons with cognitive impairment.

neuroscience↗
bioRxiv2022

Decoding of the speech envelope from EEG using the VLAAI deep neural network.

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the holdout subjects and the default test set, and only a small difference between the default test set and the public dataset was found. Compared to the baseline models, the VLAAI network still significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a logarithmic relationship between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was fine-tuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.

neuroscience↗
bioRxiv2022

Beyond Linear Neural Envelope Tracking: A Mutual Information Approach

The human brain tracks the temporal envelope of speech, which contains essential cues for speech understanding. Linear models are the most common tool to study neural envelope tracking. However, information on how speech is processed can be lost since nonlinear relations are precluded. As an alternative, mutual information (MI) analysis can detect both linear and nonlinear relations. Yet, several different approaches to calculating MI are applied without consensus on which approach to use. Furthermore, the added value of nonlinear techniques remains a subject of debate in the field. To resolve this, we applied linear and MI analyses to electroencephalography (EEG) data of participants listening to continuous speech. Comparing the different MI approaches, we conclude that results are most reliable and robust using the Gaussian copula approach, which first transforms the data to standard Gaussians. With this approach, the MI analysis is a valid technique for studying neural envelope tracking. Like linear models, it allows spatial and temporal interpretations of speech processing, peak latency analyses, and applications to multiple EEG channels combined. Finally, we demonstrate that the MI analysis can detect nonlinear components on the single-subject level, beyond the limits of linear models. We conclude that the MI analysis is a more informative tool for studying neural envelope tracking. Significance statementIn the present study, we addressed key methodological considerations for MI applications. Traditional MI methodologies require the estimation of a probability distribution at first. We show that this step can introduce a bias in the results and, consequently, severely impact interpretations. As an alternative, we propose using the parametric Gaussian copula method, which we demonstrated to be robust against biases. Second, using the parametric MI analysis, we show that there is nonlinear variance in the EEG data that the envelope of speech can explain at the single-subject level, proving its added value to neural envelope tracking. We conclude that the MI analysis is a statistically more powerful tool for studying neural envelope tracking than linear models. In addition, it retains spatial and temporal characteristics of speech processing which are lost when using more complex deep neural networks.

neuroscience↗
bioRxiv2022

Speech understanding oppositely affectsacoustic and linguistic neural tracking in aspeech rate manipulation paradigm.

When listening to continuous speech, the human brain can track features of the presented speech signal. It has been shown that neural tracking of acoustic features is a prerequisite for speech understanding and can predict speech understanding in controlled circumstances. However, the brain also tracks linguistic features of speech, which may be more directly related to speech understanding. We investigated acoustic and linguistic speech processing as a function of varying speech understanding by manipulating the speech rate. In this paradigm, acoustic and linguistic speech processing are affected simultaneously but in opposite directions: When the speech rate increases, more acoustic information per second is present. In contrast, the tracking of linguistic information becomes more challenging when speech is less intelligible at higher speech rates. We measured the EEG of 18 participants (4 male) who listened to speech at various speech rates. As expected and confirmed by the behavioral results, speech understanding decreased with increasing speech rate. Accordingly, linguistic neural tracking decreased with increasing speech rate, but acoustic neural tracking increased. This indicates that neural tracking of linguistic representations can capture the gradual effect of decreasing speech understanding. In addition, increased acoustic neural tracking does not necessarily imply better speech understanding. This suggests that, although more challenging to measure due to the low signal-to-noise ratio, linguistic neural tracking may be a more direct predictor of speech understanding. Significance statementAn increasingly popular method to investigate neural speech processing is to measure neural tracking. Although much research has been done on how the brain tracks acoustic speech features, linguistic speech features have received less attention. In this study, we disentangled acoustic and linguistic characteristics of neural speech tracking via manipulating the speech rate. A proper way of objectively measuring auditory and language processing paves the way towards clinical applications: An objective measure of speech understanding would allow for behavioral-free evaluation of speech understanding, which allows to evaluate hearing loss and adjust hearing aids based on brain responses. This objective measure would benefit populations from whom obtaining behavioral measures may be complex, such as young children or people with cognitive impairments.

neuroscience↗
bioRxiv2021

Neural tracking as an objective measure of auditory perception and speech intelligibility

When a person listens to sound, the brain time-locks to specific aspects of the sound. This is called neural tracking and it can be investigated by analysing neural responses (e.g., measured by electroencephalography) to continuous natural speech. Measures of neural tracking allow for an objective investigation of a range of auditory and linguistic processes in the brain during natural speech perception. This approach is more ecologically valid than traditional auditory evoked responses and has great potential for research and clinical applications. This article reviews the neural tracking framework and highlights three prominent examples of neural tracking analyses: neural tracking of the fundamental frequency of the voice (f0), the speech envelope and linguistic features. Each of these analyses provides a unique point of view into the human brains hierarchical stages of speech processing. F0-tracking assesses the encoding of fine temporal information in the early stages of the auditory pathway, i.e., from the auditory periphery up to early processing in the primary auditory cortex. Envelope tracking reflects bottom-up and top-down speechrelated processes in the auditory cortex and is likely necessary but not sufficient for speech intelligibility. Linguistic feature tracking (e.g. word or phoneme surprisal) relates to neural processes more directly related to speech intelligibility. Together these analyses form a multi-faceted objective assessment of an individuals auditory and linguistic processing.

neuroscience↗
bioRxiv2021

Neural markers of speech comprehension: measuring EEG tracking of linguistic speech representations, controlling the speech acoustics

When listening to speech, our brain responses time-lock to acoustic events in the stimulus. Recent studies have also reported that cortical responses track linguistic representations of speech. However, tracking of these representations is often described without controlling for acoustic properties. Therefore, the response to these linguistic representations might reflect unaccounted acoustic processing rather than language processing. Here, we evaluated the potential of several recently proposed linguistic representations as neural markers of speech comprehension. To do so, we investigated EEG responses to audiobook speech of 29 participants (22 [female]). We examined whether these representations contribute unique information over and beyond acoustic neural tracking and each other. Indeed, not all of these linguistic representations were significantly tracked after controlling for acoustic properties. However, phoneme surprisal, cohort entropy, word surprisal, and word frequency were all significantly tracked over and beyond acoustic properties. We also tested the generality of the associated responses by training on one story and testing on another. In general, the linguistic representations are tracked similarly across different stories spoken by different readers. These results suggests that these representations characterize processing of the linguistic content of speech. Significance StatementFor clinical applications it would be desirable to develop a neural marker of speech comprehension derived from neural responses to continuous speech. Such a measure would allow for behaviour-free evaluation of speech understanding; this would open doors towards better quantification of speech understanding in populations from whom obtaining behavioral measures may be difficult, such as young children or people with cognitive impairments, to allow better targeted interventions and better fitting of hearing devices.

neuroscience↗