Attention is all you need (in the brain): semantic contextualization in human hippocampus

Abstract

Word meanings in language are contextualized by surrounding words. Inspired by the self-attention mechanism in transformer-based large language models (LLMs), we hypothesized that structural composition in the brain arises from combining canonical (non-contextual) word representations with those of nearby words. We analyzed single unit activity in the human hippocampus, a region involved in semantic and contextual processing, while n=10 participants listened to podcasts. We found that hippocampal neurons encoded word position within a clause, using both ordinal and frequency-domain positional encoding. Moreover, neural responses to specific words reflected both the words own lexical semantics and a weighted sum of the embeddings of preceding words. The relative weighting of these contextualizing words correlated with LLM self-attention weights. These findings suggest that contextualization in the brain makes use of vectorial shifts that have a resemblance to attentional reweighting in LLMs, and highlight the role of the mesial temporal lobe within the broader language network.