Prioritizing replay when future goals are unknown

Abstract

Although hippocampal place cells replay nonlocal trajectories, the computational function of these events remains controversial. One hypothesis, formalized in a prominent reinforcement learning account, holds that replay plans routes to current goals. However, recent puzzling data appear to contradict this perspective by showing that replayed destinations lag current goals. These results may support an alternative hypothesis that replay updates route information to build a "cognitive map." Yet no similar theory exists to formalize this view, it is unclear how such a map is represented or what role replay plays in computing it. We address these gaps by introducing a theory of replay that learns a map of routes to candidate goals, before reward is available or when its location may change. Replay is then focused on current goals (as with planning) and/or potential future goals (like a map), depending on the animals expectations about future goal switching. Our work thus generalizes the planning account to capture a general map-building function for replay, reconciling it with data, and revealing an unexpected relationship between the seemingly distinct hypotheses. The theory offers a unifying explanation why data from tasks with different goal dynamics have seemingly supported different hypotheses for the function of replay, and suggests new predictions for experiments testing these effects. Graphical Abstract HighlightsO_LIA reinforcement learning model explains how to use replay to build a cognitive map for navigation C_LIO_LIMap-building replay explains recent replay data from goal-switching tasks that stymie previous theoretical accounts C_LIO_LIModel predicts replay in the absence of rewards as well as sensitivity of replay to goal statistics C_LIO_LIModel suggests planning-like replay is a special case of map-like replay C_LI