Biology subjects

Aubrey, B.

Publications and source records attributed to Aubrey, B..

8 recordsLinked to original sources

bioRxiv2025

The "Podcast" ECoG dataset for modeling neural activity during natural language comprehension

Naturalistic electrocorticography (ECoG) data are a rare but essential resource for studying the brains linguistic capabilities. ECoG offers a high temporal resolution suitable for investigating processes at multiple temporal timescales and frequency bands. It also provides broad spatial coverage, often along critical language areas. Here, we share a dataset of nine ECoG participants with 1,330 electrodes listening to a 30-minute audio podcast. The richness of this naturalistic stimulus can be used for various research endeavors, from auditory perception to semantic integration. In addition to the neural data, we extract linguistic features of the stimulus ranging from phonetic information to large language model word embeddings. We use these linguistic features in encoding models that relate stimulus properties to neural activity. Finally, we provide detailed tutorials for preprocessing raw data, extracting stimulus features, and running encoding analyses that can serve as a pedagogical resource or a springboard for new research.

neuroscience↗

bioRxiv2024

Information-making processes in the speaker's brain drive human conversations forward

Natural human conversation is driven by the exchange of information-rich messages that surprise the listener and deviate from predictable context. While extensive research has characterized how the brain processes unexpected linguistic input during comprehension, the neural mechanisms underlying the generation of such information by the speaker remain poorly understood. In this study, we hypothesize that the speakers brain actively creates information through extra neural computation, transforming internal thoughts into novel, meaningful linguistic outputs. Utilizing a unique 24/7 dataset of continuous electrocorticography (ECoG) comprising approximately 100 hours of spontaneous natural conversations, we contrast the neural basis of speech production and comprehension within the same participants. Behaviorally, we find that speakers take longer pausing for an additional 100-150 milliseconds before producing information-rich, improbable words, even when controlling for word identity and frequency. Neurally, we provide converging evidence for a previously unreported process: generating information-rich words elicits significantly stronger neural activity (ERP) and enhanced neural encoding and decoding in language-related areas starting 100-500 ms before articulation. This pattern contrasts sharply with speech comprehension, where enhanced neural activity for predictable words occurs before onset, while responses to improbable words emerge only after onset as prediction errors. Furthermore, we demonstrate that large language models (LLMs) mirror this biological process, requiring deeper internal computation across layers to generate improbable versus probable words. Together, these results reveal that the speakers brain is not merely a transmission channel but an active generator of information, recruiting extra neural resources to create novel content that diverges from listener expectations.

neuroscience↗

bioRxiv2024

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers havent kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a logarithmic relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

neuroscience↗

bioRxiv2024

ALIGNING BRAINS INTO A SHARED SPACE IMPROVES THEIR ALIGNMENT TO LARGE LANGUAGE MODELS

AO_SCPLOWBSTRACTC_SCPLOWRecent studies have shown that large language models (LLMs) can accurately predict neural activity measured using electrocorticography (ECoG) during natural language processing. To predict word-by-word neural activity, most prior work has estimated and evaluated encoding models within each electrode and subject--without evaluating how these models generalize across individual brains. In this paper, we analyze neural responses in 8 subjects while they listened to the same 30-minute podcast episode. We use a shared response model (SRM) to estimate a shared information space across subjects. We show that SRM significantly improves LLM-based encoding model performance. We also show that we can use this shared space to denoise the individual brain responses by projecting back into the individualized electrode space, and this process achieves a mean 38% improvement in encoding performance. The strongest improvement was observed for brain areas specialized for language comprehension, specifically in the superior temporal gyrus (STG) and inferior frontal gyrus (IFG). Critically, estimating a shared space allows us to construct encoding models that better generalize across individuals.

neuroscience↗

bioRxiv2023

Deep speech-to-text models capture the neural basis of spontaneous speech in everyday conversations

Humans effortlessly use the continuous acoustics of speech to communicate rich linguistic meaning during everyday conversations. In this study, we leverage 100 hours (half a million words) of spontaneous open-ended conversations and concurrent high-quality neural activity recorded using electrocorticography (ECoG) to decipher the neural basis of real-world speech production and comprehension. Employing a deep multimodal speech-to-text model named Whisper, we develop encoding models capable of accurately predicting neural responses to both acoustic and semantic aspects of speech. Our encoding models achieved high accuracy in predicting neural responses in hundreds of thousands of words across many hours of left-out recordings. We uncover a distributed cortical hierarchy for speech and language processing, with sensory and motor regions encoding acoustic features of speech and higher-level language areas encoding syntactic and semantic information. Many electrodes--including those in both perceptual and motor areas--display mixed selectivity for both speech and linguistic features. Notably, our encoding model reveals a temporal progression from language-to-speech encoding before word onset during speech production and from speech-to-language encoding following word articulation during speech comprehension. This study offers a comprehensive account of the unfolding neural responses during fully natural, unbounded daily conversations. By leveraging a multimodal deep speech recognition model, we highlight the power of deep learning for unraveling the neural mechanisms of language processing in real-world contexts.

neuroscience↗

bioRxiv2022

Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. Do language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language? To test this hypothesis, we densely recorded the neural activity in the Inferior Frontal Gyrus (IFG, also known as Brocas area) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derived for each patient a continuous vectorial representation for each word (i.e., a brain embedding). Using stringent, zero-shot mapping, we demonstrated that brain embeddings in the IFG and the DLM contextual embedding space have strikingly similar geometry. This shared geometry allows us to precisely triangulate the position of unseen words in both the brain embedding space (zero-shot encoding) and the DLM contextual embedding space (zero-shot decoding). The continuous brain embedding space provides an alternative computational framework for how natural language is represented in cortical language areas.

neuroscience↗

bioRxiv2020

Moment-by-moment tracking of naturalistic learning and its underlying hippocampo-cortical interactions

Every day our memory system achieves a remarkable feat: We form lasting memories of stimuli that were only encountered once. Here we investigate such learning as it naturally occurs during story listening, with the goal of uncovering when and how memories are stored and retrieved during processing of continuous, naturalistic stimuli. In behavioral experiments we confirm that, after a single exposure to a naturalistic story, participants can learn about its structure and are able to recall upcoming words in the story. In patients undergoing electrocorticographic recordings, we then track mnemonic information in high frequency activity (70 - 200Hz) as patients listen to a story twice. In auditory processing regions we demonstrate the rapid reinstatement of upcoming information after a single exposure; this neural measure of predictive recall correlates with behavioral measures of event segmentation and learning. Connectivity analyses on the neural data reveal information-flow from cortex to hippocampus at the end of events. On the second time of listening information-flow from hippocampus to cortex precedes moments of successful reinstatement.

neuroscience↗

bioRxiv2020

Thinking ahead: prediction in context as a keystone of language in humans and machines

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models are trained to generate appropriate linguistic responses in a given context. We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process natural language: 1) both are engaged in continuous next-word prediction before word-onset; 2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise (i.e., prediction error signals); 3) both represent words as a function of the previous context. In support of these three principles, our findings indicate that: a) the neural activity before word-onset contains context-dependent predictive information about forthcoming words, even hundreds of milliseconds before the words are perceived; b) the neural activity after word-onset reflects the surprise level and prediction error; and c) autoregressive DLM contextual embeddings capture the neural representation of context-specific word meaning better than arbitrary or static semantic embeddings. Together, our findings suggest that autoregressive DLMs provide a novel and biologically feasible computational framework for studying the neural basis of language.

neuroscience↗