Biology subjects

Mesgarani, N.

Publications and source records attributed to Mesgarani, N..

At least 19 recordsLinked to original sources

bioRxiv2026

Neural Decoding of Musical Engagement Reveals Tension-Release Dynamics

Music listening is one of the most compelling and rewarding activities humans engage in spontaneously. But what exactly catches people"s attention when listening to music remains unclear. Musicologists have argued that tension-release dynamics in music constitute crucial features of the listening experience. They arise from the intertwining of different low-level and high-level features and sit at the core of music enjoyment by engaging listeners dynamically. This study aims to characterize the relationship between tension dynamics and engagement during naturalistic music listening. Using canonical correlation analysis, we decoded the music envelope from EEG and ECoG responses and found that musical tension patterns, as reported by listeners, were predictive of fluctuations in the coupling between the music and the neural response. Importantly, tension dynamics were significantly correlated with neural measures of envelope tracking even after controlling for loudness and musical expectations, confirming the specific and crucial role of musical tension in engaging listeners. These results shed new light on how musical structure gives rise to internal response modulations that, in turn, dynamically reflect musical engagement. This interplay may underlie the pervasive and emotionally rewarding nature of music. Significance StatementThis work addresses a fundamental question in cognition: how engagement dynamically modulates complex auditory input processing. We investigate this issue within the well-controlled yet ecologically valid context of music perception. At the core of the musical experience lies the perception of tension - the shifting sense of expectancy, instability, and resolution that guides listening across time. We show that fluctuations in musical tension, shaped by musical structure, reliably predict modulations of auditory engagement, thereby illuminating an understudied phenomenon: what makes music engaging.

neuroscience↗

bioRxiv2025

Speaker Identity is Robustly Encoded in Spatial Patterns of Intracranial EEG for Attention Decoding

The human auditory cortex robustly tracks attended speech, yet it remains unclear if speaker identity is encoded in spatial patterns of neural activity independent of temporal dynamics. Here, we demonstrate that the identity of an attended speaker is reliably reflected in distinct, time-invariant spatial activation maps in human intracranial EEG (iEEG). Leveraging these "neural fingerprints", we developed a novel framework for Auditory Attention Decoding (AAD) that shifts from traditional temporal envelope tracking to spatial speaker identification. By decoupling the decoding of "who" is speaking from "when" they are speaking, our modular system achieves state-of-the-art speech extraction, particularly in short time windows (<2 seconds) where temporal models typically fail. Furthermore, we observed a reciprocal shift in neural activity during attentional switches, confirming that these spatial codes dynamically track listener intent. These findings establish that speaker identity is a robust, spatially distributed feature in the auditory cortex, offering a high-speed, complementary mechanism for neuro-steered hearing technologies.

neuroscience↗

bioRxiv2025

Evaluating scientific theories as predictive models in language neuroscience

Modern data-driven encoding models are highly effective at predicting brain responses to language stimuli. However, these models struggle to explain the underlying phenomena, i.e. what features of the stimulus drive the response? We present Question Answering encoding models, a method for converting qualitative theories of language selectivity into highly accurate, interpretable models of brain responses. QA encoding models annotate a language stimulus by using a large language model to answer yes-no questions corresponding to qualitative theories. A compact QA encoding model that uses only 35 questions outperforms existing baselines at predicting brain responses in both fMRI and ECoG data. The model weights also provide easily interpretable maps of language selectivity across cortex; these maps show quantitative agreement with meta-analyses of the existing literature and selectivity maps identified in a follow-up fMRI experiment. These results demonstrate that LLMs can bridge the widening gap between qualitative scientific theories and data-driven models.

neuroscience↗

bioRxiv2025

Large Language Models Reveal the Neural Tracking of Linguistic Context in Attended and Unattended Multi-Talker Speech

Large language models (LLMs) capture long-range contextual structure in natural language and have recently been shown to align with the human brains contextualized linguistic encoding. This makes them a promising computational probe for studying how context-dependent linguistic information is represented during natural speech perception. Speech perception often occurs in multi-talker environments, where attention must dynamically select among competing streams, yet how contextual information from attended and unattended speech is neurally encoded remains underexplored. Here, we investigate how auditory attention modulates neural tracking of context-dependent linguistic representations using electrocorticography (ECoG) and stereoelectroencephalography (sEEG) recordings from three epilepsy patients engaged in a two-conversation "cocktail party" paradigm. To model neural responses to attended and unattended speech streams, we used contextual word embeddings generated by large language models. We find that LLM-derived features reliably predict brain activity for the attended stream and that contextual information from the unattended stream also contributes to neural prediction. Importantly, these contributions extend beyond low-level acoustic features and shallow syntactic information, and depend on the surrounding linguistic context. Moreover, neural tracking of the unattended stream reflects shorter-range contextual integration than that of the attended stream. Together, these findings indicate that neural responses to speech reflect context-dependent linguistic representations from multiple concurrent speech streams, with attention modulating the depth and timescale of contextual integration. Our results highlight the utility of LLMs for probing higher-level linguistic representations in complex, naturalistic listening environments.

neuroscience↗

bioRxiv2025

Parallel hierarchical encoding of linguistic representations in the human auditory cortex and recurrent automatic speech recognition systems

Transforming continuous acoustic speech signals into discrete linguistic meaning is a remarkable computational feat accomplished by both the human brain and modern artificial intelligence. A key scientific question is whether these biological and artificial systems, despite their different architectures, converge on similar strategies to solve this challenge. While ASR systems now achieve human-level performance, research on their parallels with the brain has been limited by biologically implausible, non-causal models and comparisons that stop at predicting brain activity without detailing the alignment of the underlying representations. Furthermore, studies using text-based models overlook the crucial acoustic stages of speech processing. Here, using high-resolution intracranial recordings and a causal, recurrent ASR model, we bridge these gaps by uncovering a striking correspondence between the brains processing hierarchy and the models internal representations. Specifically, we demonstrate a deep alignment in their algorithmic approach: neural activity in distinct cortical regions maps topographically to corresponding model layers, and critically, the representational content at each stage follows a parallel progression from acoustic to phonetic, lexical, and semantic information. This work thus moves beyond demonstrating simple model-brain alignment to specifying the shared underlying representations at each stage of processing, providing direct evidence that both systems converge on a similar computational strategy for transforming sound into meaning.

neuroscience↗

bioRxiv2024

The impact of musical expertise on disentangled and contextual neural encoding of music revealed by generative music models

Music perception involves the intricate processing of individual notes and their contextual relationships within a piece. However, how the brain encodes and organizes these features, particularly in relation to musical expertise, remains unclear. Using noninvasive and invasive electrophysiological recordings alongside generative music models, we reveal that musicians exhibit neural encoding that is more attuned to the separation and integration of complex musical structures, with a pronounced left-hemispheric bias compared to non-musicians. Invasive recordings further highlight a hierarchical and spatially organized representation of musical features and context across brain regions. These findings advance our understanding of how the brain processes music and the role of musical training in shaping auditory cognition.

neuroscience↗

bioRxiv2024

Distinct roles of SNR, speech Intelligibility, and attentional effort on neural speech tracking in noise

Robust neural encoding of speech in noise is influenced by several factors, including signal-to-noise ratio (SNR), speech intelligibility (SI), and attentional effort (AE). Yet, the interaction and distinct role of these factors remain unclear. In this study, fourteen native English speakers performed selective speech listening tasks at various SNR levels while EEG responses were recorded. Attentional performance was assessed using a repeated word detection task, and attentional effort was inferred from subjects gaze velocity. Results indicate that both SNR and SI enhance neural tracking of target speech, with distinct effects influenced by the previously overlooked role of attentional effort. Specifically, at high levels of SI, increasing SNR leads to reduced attentional effort, which in turn decreases neural speech tracking. Our findings highlight the importance of differentiating the roles of SNR, SI, and AE in neural speech processing and advance our understanding of how noisy speech is processed in the auditory pathway.

neuroscience↗

bioRxiv2024

Decoding the Unintelligible: Neural Speech Tracking in Low Signal-to-Noise Ratios

Understanding speech in noisy environments is challenging for both human listeners and speech technologies, with significant implications for hearing aid design and communication systems. Auditory attention decoding (AAD) aims to decode the attended talker from neural signals to enhance their speech and improve perception. However, whether this decoding remains reliable under severely degraded listening conditions remains unclear. In this study, we investigated selective neural tracking of the attended speaker under adverse listening conditions. Using EEG recordings in a multi-talker speech perception task with varying SNR, participants task performance--quantified through a repeated-word detection task--was analyzed as a proxy for perceptual accuracy and attentional focus, while neural responses were used to decode the attended talker. Despite substantial degradation in task performance, we found that neural tracking of attended speech persists, suggesting that the brain retains sufficient information for decoding. These findings demonstrate that even in highly challenging conditions, AAD remains feasible, offering a potential avenue for improving speech perception in brain-informed audio technologies, such as hearing aids, that leverage AAD to enhance listening experiences in real-world noisy environments.

neuroscience↗

bioRxiv2024

Temporal integration in human auditory cortex is predominantly yoked to absolute time, not structure duration

Sound structures such as phonemes and words have highly variable durations. Thus, there is a fundamental difference between integrating across absolute time (e.g., 100 ms) vs. sound structure (e.g., phonemes). Auditory and cognitive models have traditionally cast neural integration in terms of time and structure, respectively, but the extent to which cortical computations reflect time or structure remains unknown. To answer this question, we rescaled the duration of all speech structures using time stretching/compression and measured integration windows in the human auditory cortex using a new experimental/computational method applied to spatiotemporally precise intracranial recordings. We observed significantly longer integration windows for stretched speech, but this lengthening was very small ([~]5%) relative to the change in structure durations, even in non-primary regions strongly implicated in speech-specific processing. These findings demonstrate that time-yoked computations dominate throughout the human auditory cortex, placing important constraints on neurocomputational models of structure processing.

neuroscience↗

bioRxiv2024

Joint population coding and temporal coherence link an attended talker's voice and location features in naturalistic multi-talker scenes.

Listeners effortlessly extract multidimensional auditory objects, such as a localized talker, from complex acoustic scenes. However, the neural mechanisms that enable simultaneous encoding and linking of distinct sound features--such as a talkers voice and location--are not fully understood. Using invasive intracranial recordings in neurosurgical patients, we investigated how the human auditory cortex processes and integrates these features during naturalistic multi-talker scenes. We found that cortical sites exhibit a gradient of feature sensitivity, ranging from single-feature sensitive sites (responsive primarily to voice or location) to dual-feature sensitive sites (responsive to both features). At the population level, neural response patterns from both single- and dual-feature sensitive sites jointly encoded the attended talkers voice and location. Notably, single-feature sensitive sites encoded their primary feature with greater precision but also represented coarse information about the secondary feature. Sites selectively tracking a single, attended speech stream concurrently encoded both voice and location features, demonstrating a link between selective attention and feature integration. Additionally, attention selectively enhanced temporal coherence between voice- and location-sensitive sites, suggesting that temporal synchronization serves as a mechanism for linking these features. Our findings highlight two complementary neural mechanisms--joint population coding and temporal coherence-- that enable the integration of voice and location features in the auditory cortex. These results provide new insights into the distributed, multidimensional nature of auditory object formation during active listening in complex environments. SIGNIFICANCE STATEMENTIn everyday life, listeners effortlessly extract individual sound sources from complex acoustic scenes which contain multiple sound sources. Yet, how the brain links the different features of a particular sound source to each other - such as a talkers voice characteristics and location - is poorly understood. Here, we show that two neural mechanisms contribute to encoding and integrating voice and location features in multi-talker sound scenes: (1) some neuronal sites are sensitive to both voice and location and their activity patterns encode these features jointly; (2) the responses of neuronal sites that process only one sound feature - that is, location or voice - align temporally to form a stream that is segregated from the other talker. HIGHLIGHTSO_LIAuditory cortex exhibits a gradient of feature sensitivity, with some sites encoding only voice or location features, while others encode both simultaneously (dual-feature sensitive sites). C_LIO_LIDual-feature sensitive sites integrate voice and location features of an attended talker with equal accuracy, providing a unified representation in multi-talker scenes. C_LIO_LISingle-feature sensitive sites primarily encode their preferred feature with high precision but also represent coarse information about other features, contributing to population-level integration. C_LIO_LITemporal coherence selectively enhances synchronization between voice- and location-sensitive sites, providing another mechanism for integrating an auditory objects features. C_LIO_LIMulti-dimensional auditory object formation relies on complementary neural mechanisms: joint population coding and temporal coherence. C_LI

neuroscience↗

bioRxiv2024

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Focusing on a specific conversation amidst multiple interfering talkers presents a significant challenge, especially for the hearing-impaired. Brain-controlled assistive hearing devices aim to alleviate this problem by separating complex auditory scenes into distinct speech streams and enhancing the attended speech based on the listeners neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, we present a realistic AAD task that mirrors the dynamic nature of acoustic settings. This task involves focusing on one of two concurrent conversations, with multiple talkers taking turns and moving continuously in space with background noise. Invasive electroencephalography (iEEG) data were collected from three neurosurgical patients as they focused on one of the two moving conversations. We propose an enhanced brain-controlled assistive hearing system that combines AAD and a binaural speaker-independent speech separation model. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve auditory attention decoding accuracy. Our subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of our approach in real-world scenarios and marks a significant step towards developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences. TAKEAWAYS- Brain-controlled hearing device for scenarios with moving conversations in multi-talker settings, closely mimicking real-world listening environments - Developed a binaural speech separation model that separates speech of moving talkers while retaining their spatial locations, enhancing auditory perception and auditory attention decoding - Proposed system enhances speech intelligibility and reduces listening effort in realistic acoustic scenes

neuroscience↗

bioRxiv2021

The Spatial Reach of Neuronal Coherence and Spike-field Coupling across the Human Neocortex

Neuronal coherence is thought to be a fundamental mechanism of communication in the brain, where synchronized field potentials coordinate synaptic and spiking events to support plasticity and learning. Although the spread of field potentials has garnered great interest, little is known about the spatial reach of phase synchronization, or neuronal coherence. Functional connectivity between different brain regions is known to occur across long distances, but the locality of coherence within a brain region is understudied. Here we used simultaneous recordings from electrocorticography (ECoG) grids and high-density microelectrode arrays to estimate the spatial reach of neuronal coherence and spike-field coherence (SFC) across frontal, temporal, and occipital cortices during cognitive tasks in humans. We observed the strongest coherence within a 2-3 cm distance from the microelectrode arrays, potentially defining an effective range for local communication. This range was relatively consistent across brain regions, spectral frequencies, and cognitive tasks. The magnitude of coherence showed power law decay with increasing distance from the microelectrode arrays, where the highest coherence occurred between ECoG contacts, followed by coherence between ECoG and deep cortical LFP, and then SFC (i.e., ECoG > LFP > SFC). The spectral frequency of coherence also affected its magnitude. Alpha coherence (8-14 Hz) was generally higher than other frequencies for signals nearest the microelectrode arrays, whereas delta coherence (1-3 Hz) was higher for signals that were farther away. Action potentials in all brain regions were most coherent with the phase of alpha oscillations, which suggests that alpha waves could play a larger, more spatially local role in spike timing than other frequencies. These findings provide a deeper understanding of the spatial and spectral dynamics of neuronal coherence, further advancing knowledge about how activity propagates across the human brain.

neuroscience↗

bioRxiv2020

Hierarchical integration across multiple timescales in human auditory cortex

To derive meaning from sound, the brain must integrate information across tens (e.g. phonemes) to hundreds (e.g. words) of milliseconds, but the neural computations that enable multiscale integration remain unclear. Prior evidence suggests that human auditory cortex analyzes sound using both generic acoustic features (e.g. spectrotemporal modulation) and category-specific computations, but how these putatively distinct computations integrate temporal information is unknown. To answer this question, we developed a novel method to estimate neural integration periods and applied the method to intracranial recordings from human epilepsy patients. We show that integration periods increase three-fold as one ascends the auditory cortical hierarchy. Moreover, we find that electrodes with short integration periods (~50-150 ms) respond selectively to spectrotemporal modulations, while electrodes with long integration periods (~200-300 ms) show prominent selectivity for sound categories such as speech and music. These findings reveal how multiscale temporal analysis organizes hierarchical computation in human auditory cortex.

neuroscience↗

bioRxiv2020

Does the phase of ongoing EEG oscillations predict auditory perception?

Effective processing of information from the environment requires the brain to selectively sample relevant inputs. The visual perceptual system has been shown to sample information rhythmically, oscillating rapidly between more and less input-favorable states. Evidence of parallel effects in auditory perception is inconclusive. Here, we combined a bilateral pitch-identification task with electroencephalography (EEG) to investigate whether the phase of ongoing EEG predicts auditory discrimination accuracy. We compared prestimulus phase distributions between correct and incorrect trials. Shortly before stimulus onset, each of these distributions showed significant phase concentration, but centered at different phase angles. The effects were strongest in theta and beta frequency bands. The divergence between phase distributions showed a linear relation with accuracy, accounting for at least 10% of inter-individual variance. Discrimination performance oscillated rhythmically at a rate predicted by the neural data. These findings indicate that auditory discrimination threshold oscillates over time along with the phase of ongoing EEG activity. Thus, it appears that auditory perception is discrete rather than continuous, with the phase of ongoing EEG oscillations shaping auditory perception by providing a temporal reference frame for information processing.

neuroscience↗

bioRxiv2020

Functional characterization of human Heschl's gyrus in response to natural speech

Heschls gyrus (HG) is a brain area that includes the primary auditory cortex in humans. Due to the limitations in obtaining direct neural measurements from this region during naturalistic speech listening, the functional organization and the role of HG in speech perception remains uncertain. Here, we used intracranial EEG to directly record the neural activity in HG in eight neurosurgical patients as they listened to continuous speech stories. We studied the spatial distribution of acoustic tuning and the organization of linguistic feature encoding. We found a main gradient of change from posteromedial to anterolateral parts of HG. Along this direction, we observed a decrease in frequency and temporal modulation tuning, and an increase in phonemic representation, speaker normalization, speech-sensitivity, and response latency. We did not observe a difference between the two brain hemispheres. These findings reveal a functional role for HG in processing and transforming simple to complex acoustic features and informs neurophysiological models of speech processing in the human auditory cortex.

neuroscience↗

bioRxiv2020

Neural representation of linguistic feature hierarchy reflects second-language proficiency

Acquiring a new language requires a simultaneous and gradual learning of multiple levels of linguistic attributes. Here, we investigated how this process changes the neural encoding of natural speech by assessing the encoding of the linguistic feature hierarchy in second-language listeners. Electroencephalography (EEG) signals were recorded during English story listening from native Mandarin speakers with varied English proficiency and from native English speakers. We measured the temporal response functions (TRF) for acoustic, phonemic, phonotactic, and semantic features in individual participants and found a main effect of proficiency on the linguistic encoding. This effect of second-language proficiency was particularly prominent on the neural encoding of phonemes, showing stronger encoding of "new" phonemic contrasts (i.e. English contrasts that do not exist in Mandarin) with increasing proficiency. Overall, we found that linguistic feature representation in nonnative listeners progressively converged to that of native listeners with proficiency, which enabled accurate decoding of language proficiency. This detailed view advances our understanding of the cortical processing of linguistic information in second-language learners and provides an objective measure of language proficiency.

neuroscience↗

bioRxiv2019

Estimating and interpreting nonlinear receptive fields of sensory responses with deep neural network models

Sensory processing by neural circuits includes numerous nonlinear transformations that are critical to perception. Our understanding of these nonlinear mechanisms, however, is hindered by the lack of a comprehensive and interpretable computational framework that can model and explain nonlinear signal transformations. Here, we propose a data-driven framework based on deep neural network regression models that can directly learn any nonlinear stimulus-response mapping. A key component of this approach is an analysis method that reformulates the exact function of the trained neural network as a collection of stimulus-dependent linear functions. This locally linear receptive field interpretation of the network function enables straightforward comparison with conventional receptive field models and uncovers nonlinear encoding properties. We demonstrate the efficacy of this framework by predicting the neural responses recorded invasively from the auditory cortex of neurosurgical patients as they listened to speech. Our method significantly improves the prediction accuracy of auditory cortical responses particularly in nonprimary areas. Moreover, interpreting the functions learned by neural networks uncovered three distinct types of nonlinear transformations of speech that varied considerably in primary and nonprimary auditory regions. By combining two desired properties of a computational sensory-response model; the ability to capture arbitrary stimulus-response mappings and maintaining model interpretability, this data-driven method can lead to better neurophysiological models of the sensory processing.

neuroscience↗

bioRxiv2019

Cortical encoding of melodic expectations in human temporal cortex

Humans engagement in music rests on underlying elements such as the listeners cultural background and general interest in music, all shaping the way music is processed in the brain and perceived. Crucially, these factors modulate how listeners anticipate musical events, a process inducing instantaneous neural responses as the music confronts these expectations. Measuring such neural correlates would represent a direct window into high-level brain processing of music. Here we recorded electroencephalographic and electrocorticographic brain responses as participants listened to Bach melodies. We assessed the relative contributions of the acoustic versus melodic components of the music to the neural signal. Acoustic features included envelope and its derivative. Melodic features included information on melodic progressions (pitch) and their tempo (onsets), which were extracted from a Markov model predicting the next note based on a corpus of Western music and the preceding proximal musical context. We related the music to brain activity with a linear temporal response function, and demonstrated that cortical responses to music encode melodic expectations. Specifically, individual-subject neural signals were better predicted by a combination of acoustic and melodic expectation features than by either alone. This effect was most pronounced at response latencies up to 350ms, and in both planum temporale and Heschls gyrus. Finally, expectations of pitch and onset-time of musical notes exerted independent cortical effects, and such influences were modulated by the listeners musical expertise. Overall, this study demonstrates how the interplay of experimental and theoretical approaches can yield novel insights into the cortical encoding of melodic expectations.

neuroscience↗