Griffiths, T. L. | bioRxiv Science

bioRxiv2022

Modeling human eye movements during immersive visual search

The nature of eye movements during visual search has been widely studied in psychology and neuroscience. Virtual reality (VR) paradigms provide an opportunity to test whether computational models of search can predict naturalistic search behavior. However, existing ideal observer models are constrained by strong assumptions about the structure of the world, rendering them impractical for modeling the complexity of environments that can be studied in VR. To address these limitations, we frame naturalistic visual search as a problem of allocating limited cognitive resources, formalized as a meta-level Markov decision process (meta-MDP) over a representation of the environment encoded by a deep neural network. We train reinforcement learning agents to solve the meta-MDP, showing that the agents optimal policy converges to a classic ideal observer model of search developed for simplified environments. We compare the learned policy with human gaze data from a visual search experiment conducted in VR, finding a qualitative and quantitative correspondence between model predictions and human behavior. Our results suggest that gaze behavior in naturalistic visual search is consistent with rational allocation of limited cognitive resources.

neuroscience↗

bioRxiv2022

Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model

Humans use complex linguistic structures to transmit ideas to one another. The brain is thought to deploy specialized computations to process these structures. Recently, a new class of artificial neural networks based on the Transformer architecture has revolutionized the field of language modeling, attracting attention from neuroscientists seeking to understand the neurobiology of language in silico. Transformers integrate information across words via multiple layers of structured circuit computations, forming increasingly contextualized representations of linguistic content. Prior work has focused on the internal representations (the "embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into functionally-specialized "transformations" to provide a complementary window onto linguistic computations in the human brain. Using functional MRI data acquired while participants listened to naturalistic spoken stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent syntactic computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers, contextual distances, and syntactic dependencies in a low-dimensional cortical space. Our findings indicate that large language models and the cortical language network may converge on similar trends of functional specialization for processing natural language.

neuroscience↗