Biology subjects

Huth, A. G.

Publications and source records attributed to Huth, A. G..

14 recordsLinked to original sources

bioRxiv2026

The human language processing system straightens natural speech

Large language models trained on next-word prediction have impressive linguistic capabilities. This suggests that the goal of temporal prediction is essential to language processing, but how this goal impacts the structure of speech representations in the human brain remains unknown. Here, we test the hypothesis that prediction is facilitated by the temporal straightening of representational trajectories along the speech processing hierarchy. We developed a methodology for measuring the curvature of these trajectories using fMRI. Our method exploits a previously unknown connection between the timescale of single-unit responses and the curvature of population trajectories. We examined brain responses of subjects listening to natural speech. Response trajectories were most curved in lower-level auditory areas and progressively straightened along the cortical hierarchy. We presented the same speech stimuli and perturbed versions thereof to wavLM--a speech representation model that is well aligned with human brain responses--and found that hierarchical straightening effects are strongest for stimuli whose statistical structure resembles natural speech. Together, our results establish a direct connection between the goal of temporal prediction, the geometry of neural speech representations, and the cortical hierarchy of representational timescales.

neuroscience↗

bioRxiv2026

Decoding concept representations in aphasia after stroke

Many stroke survivors with aphasia struggle to map their thoughts into words or motor plans. Neuroprostheses that decode concept representations could help these individuals communicate by predicting the words, phrases, or sentences that they are struggling to produce. Here we decoded concept representations measured using functional magnetic resonance imaging (fMRI) from participants with different aphasia profiles. The decoders generated continuous word sequences that could describe the concepts that the participants were hearing about, seeing, or imagining. To forecast how this approach would generalize across the heterogeneity of aphasia profiles, we characterized how stroke affects the anatomical organization and information capacity of conceptual processing. Mapping how concepts are organized across the brain, we found that conceptual tuning during non-linguistic processing was largely consistent between the participants with aphasia and neurologically healthy participants. Comparing information processing between the participants with aphasia and neurologically healthy participants, we found that both groups processed similar amounts of non-linguistic information. Our findings indicate that concept representations can be largely spared in individuals with aphasia and demonstrate how these representations can be decoded to support communication.

neuroscience↗

bioRxiv2026

Rapid cortical mapping with cross-participant encoding models

Voxelwise encoding models trained on functional MRI data can produce detailed maps of cortical organization. However, voxelwise encoding models must be trained on many hours of brain responses from each participant, limiting clinical applications. In this study, we introduce a cross-participant modeling framework for rapid cortical mapping. In this framework, voxelwise encoding models are trained on many hours of brain responses from previously scanned reference participants, and then transferred to a new participant by aligning brain responses using a small set of stimuli. We evaluated cross-participant encoding models on linguistic semantic mapping, non-linguistic semantic mapping, and auditory mapping. In each case we found that cross-participant encoding models had more accurate selectivity estimates and prediction performance than within-participant encoding models trained on the same amount of data from the new participant. We also found that cross-participant encoding models improved with the amount of data from each reference participant and the number of reference participants. These results demonstrate that cross-participant modeling can substantially reduce the amount of data required for accurate cortical mapping, which may facilitate new clinical applications of functional neuroimaging.

neuroscience↗

bioRxiv2025

Visual Semantic Encoding and Identification of Naturalistic Movies via High-Density Diffuse Optical Tomography

Understanding how the brain represents meaning in real-world contexts is essential for both fundamental neuroscience and clinical applications. Brain encoding and decoding models from naturalistic stimuli provide a powerful window into semantic representations. Yet, existing approaches rely on a constrained scanning environment, or on conventional fNIRS, which has been limited to sparse sampling and/or block-design paradigms. Here, we tested whether high-density diffuse optical tomography (HD-DOT), an advanced high-density tomographic optical imaging method, can support semantic encoding and decoding using naturalistic movies. We collected 3.5 hours of naturalistic movie viewing data from six participants using stimuli labeled with 1,708 categories. Encoding models robustly predicted voxel-level responses, yielding single semantic category maps consistent with prior fMRI studies. In complementary decoding analyses, we showed that DOT responses captured sufficient semantic content to identify which clips participants viewed. To assess organization across individuals, we identified a shared low-dimensional semantic space that captures common semantic dimensions. Finally, clustering analyses revealed interpretable higher-order semantic dimensions like social and animate agents, objects vs natural organisms, and textural scenes, consistently mapped across the cortex. These findings demonstrate that DOT can recover distributed, high-dimensional semantic representations from naturalistic movies, bridging fMRI-level semantic mapping with the accessibility of optical imaging.

neuroscience↗

bioRxiv2025

Evaluating scientific theories as predictive models in language neuroscience

Modern data-driven encoding models are highly effective at predicting brain responses to language stimuli. However, these models struggle to explain the underlying phenomena, i.e. what features of the stimulus drive the response? We present Question Answering encoding models, a method for converting qualitative theories of language selectivity into highly accurate, interpretable models of brain responses. QA encoding models annotate a language stimulus by using a large language model to answer yes-no questions corresponding to qualitative theories. A compact QA encoding model that uses only 35 questions outperforms existing baselines at predicting brain responses in both fMRI and ECoG data. The model weights also provide easily interpretable maps of language selectivity across cortex; these maps show quantitative agreement with meta-analyses of the existing literature and selectivity maps identified in a follow-up fMRI experiment. These results demonstrate that LLMs can bridge the widening gap between qualitative scientific theories and data-driven models.

neuroscience↗

bioRxiv2025

Efficient uniform sampling explains non-uniform memory of narrative stories

Humans do not remember all experiences uniformly. We remember certain moments better than others, and central gist better than detail. Current theories focus exclusively on surprise to explain why some moments are better remembered, and do not explain gist memory. We propose that humans uniformly sample incoming information in time, which explains both non-uniform memory and gist. Rather than surprise, this model predicts that the mutual information between a given moment and the rest of the experience drives memory. To test this model, participants listened to narrative stories and recalled them immediately afterward. Using large language models to quantify the information structure of narrative stories and participants recall, we found that our parsimonious uniform sampling model explained memory better than earlier theories. These findings suggest an alternative, simpler account of human memory that does not rely on costly feedback mechanisms for prioritizing encoding of specific information.

neuroscience↗

bioRxiv2023

A unifying computational account of temporal context effects in language across the human cortex

Deep learning advances have revolutionized computational modeling approaches in neuroscience. However, their black-box nature makes it challenging to use deep learning models to discover new insights about brain function. Focusing on human language processing, we propose a new framework to improve the quality and interpretability of the inferences we make from deep learning-based models. First, we add interpretable components to a deep language model and use it to build a predictive encoding model. Then, we use the models predictive abilities to simulate brain responses to controlled stimuli from published experiments. We find that our model, based on a multi-timescale recurrent neural network, captures many previously reported temporal context effects in human cortex. Its failure to capture other effects also highlights important gaps in current language models. Finally, we use this new framework to generate model-based evidence that supports the proposal that different linguistic features are represented at different timescales across cortex.

neuroscience↗

bioRxiv2022

Semantic reconstruction of continuous language from non-invasive brain recordings

A brain-computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, decoders that reconstruct continuous language use invasive recordings from surgically implanted electrodes1-3, while decoders that use non-invasive recordings can only identify stimuli from among a small set of letters, words, or phrases4-7. Here we introduce a non-invasive decoder that reconstructs continuous natural language from cortical representations of semantic meaning8 recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech, and even silent videos, demonstrating that a single language decoder can be applied to a range of semantic tasks. To study how language is represented across the brain, we tested the decoder on different cortical networks, and found that natural language can be separately decoded from multiple cortical networks in each hemisphere. As brain-computer interfaces should respect mental privacy9, we tested whether successful decoding requires subject cooperation, and found that subject cooperation is required both to train and to apply the decoder. Our study demonstrates that continuous language can be decoded from non-invasive brain recordings, enabling future multipurpose brain-computer interfaces.

neuroscience↗

bioRxiv2022

A natural language fMRI dataset for voxelwise encoding models

Speech comprehension is a complex process that draws on humans abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 subjects each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.

neuroscience↗

bioRxiv2021

Cortical Representations of Concrete and Abstract Concepts in Language Combine Visual and Linguistic Representations

The human semantic system stores knowledge acquired through both perception and language. To study how semantic representations in cortex integrate perceptual and linguistic information, we created semantic word embedding spaces that combine models of visual and linguistic processing. We then used these visually-grounded semantic spaces to fit voxelwise encoding models to fMRI data collected while subjects listened to hours of narrative stories. We found that cortical regions near the visual system represent concepts by combining visual and linguistic information, while regions near the language system represent concepts using mostly linguistic information. Assessing individual representations near visual cortex, we found that more concrete concepts contain more visual information, while even abstract concepts contain some amount of visual information from associated concrete concepts. Finally we found that these visual grounding effects are localized near visual cortex, suggesting that semantic representations specifically reflect the modality of adjacent perceptual systems. Our results provide a computational account of how visual and linguistic information are combined to represent concrete and abstract concepts across cortex.

neuroscience↗

bioRxiv2021

Voxelwise encoding models show that cerebellar language representations are highly conceptual

There is a growing body of research demonstrating that the cerebellum is involved in language understanding. Early theories assumed that the cerebellum is involved in low-level language processing. However, those theories are at odds with recent work demonstrating cerebellar activation during cognitive tasks. Using natural language stimuli and an encoding model framework, we performed an fMRI experiment where subjects passively listened to five hours of natural language stimuli which allowed us to analyze language processing in the cerebellum with higher precision than previous work. We used this data to fit voxelwise encoding models with five different feature spaces that span the hierarchy of language processing from acoustic input to high-level conceptual processing. Examining the prediction performance of these models on separate BOLD data shows that cerebellar responses to language are almost entirely explained by high-level conceptual language features rather than low-level acoustic or phonemic features. Additionally, we found that the cerebellum has a higher proportion of voxels that represent social semantic categories, which include "social" and "people" words, and lower representations of all other semantic categories, including "mental", "concrete", and "place" words, than cortex. This suggests that the cerebellum is representing language at a conceptual level with a preference for social information. Significance StatementRecent work has demonstrated that, beyond its typical role in motor planning, the cerebellum is implicated in a wide variety of tasks including language. However, little is known about the language representations in the cerebellum, or how those representations compare to cortex. Using voxelwise encoding models and natural language fMRI data, we demonstrate here that language representations are significantly different in the cerebellum as compared to cortex. Cerebellum language representations are almost entirely semantic, and the cerebellum contains over-representation of social semantic information as compared to cortex. These results suggest that the cerebellum is not involved in language processing per se, but cognitive processing more generally.

neuroscience↗

bioRxiv2020

Attentional Modulation of Hierarchical Speech Representations in a Multi-Talker Environment

Humans are remarkably adept in listening to a desired speaker in a crowded environment, while filtering out non-target speakers in the background. Attention is key to solving this difficult cocktail-party task, yet a detailed characterization of attentional effects on speech representations is lacking. It remains unclear across what levels of speech features and how much attentional modulation occurs in each brain area during the cocktail-party task. To address these questions, we recorded whole-brain BOLD responses while subjects either passively listened to single-speaker stories, or selectively attended to a male or a female speaker in temporally-overlaid stories in separate experiments. Spectral, articulatory, and semantic models of the natural stories were constructed. Intrinsic selectivity profiles were identified via voxelwise models fit to passive listening responses. Attentional modulations were then quantified based on model predictions for attended and unattended stories in the cocktail-party task. We find that attention causes broad modulations at multiple levels of speech representations while growing stronger towards later stages of processing, and that unattended speech is represented up to the semantic level in parabelt auditory cortex. These results provide insights on attentional mechanisms that underlie the ability to selectively listen to a desired speaker in noisy multi-speaker environments.

neuroscience↗

bioRxiv2020

Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech

Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. Further, the standard fMRI encoding procedure does not account for varying temporal properties in the encoding features. We modify the procedure so that it can capture both short- and long-timescale information. This approach outperforms other encoding models, particularly for voxels that represent long-timescale information. It also provides a finer-grained map of timescale information in the human language pathway. This serves as a framework for future work investigating temporal hierarchies across artificial and biological language systems.

neuroscience↗

bioRxiv2018

Voxelwise encoding models with non-spherical multivariate normal priors

Predictive models for neural or fMRI data are often fit using regression methods that employ priors on the model parameters. One widely used method is ridge regression, which employs a spherical Gaussian prior that assumes equal and independent variance for all parameters. However, a spherical prior is not always optimal or appropriate. There are many cases where expert knowledge or hypotheses about the structure of the model parameters could be used to construct a better prior. In these cases, non-spherical Gaussian priors can be employed using a generalized form of ridge known as Tikhonov regression. Yet Tikhonov regression is only rarely used in neuroscience. In this paper we discuss the theoretical basis for Tikhonov regression, demonstrate a computationally efficient method for its application, and show several examples of how Tikhonov regression can improve predictive models for fMRI data. We also show that many earlier studies have implicitly used Tikhonov regression by linearly transforming the regressors before performing ridge regression.

neuroscience↗