Mixture of Learning Strategies Underlies Rodent Behavior in Dynamic Foraging

Abstract

In volatile foraging environments, animals need to adapt their learning in accordance with the uncertainty of the environment and knowledge of the hidden structure of the world. In these contexts, previous studies have distinguished between two types of strategies, model-free learning, where reward values are updated locally based on external feedback signals, and inference-based learning, where an internal model of the world is used to make optimal inferences about the current state of the environment. Distinguishing between these strategies during the dynamic foraging behavioral paradigm has been a challenging problem for studies of reward-guided decisions, due to the diversity in behavior of model-free and inference-based agents, as well as the complexities that arise when animals mix between these types of strategies. Here, we developed two solutions that jointly tackle these problems. First, we identified four key behavioral features that together benchmark the switching dynamics of agents in response to a change in reward contingency. We performed computational simulations to systematically measure these features for a large ensemble of model-free and inference-based agents, uncovering an organized structure of behavioral choices where observed behavior can be reliably classified into one of six distinct regimes in the two respective parameter spaces. Second, to address the challenge that arises when animals use multiple strategies within single sessions, we developed a novel state-space method, block Hidden Markov Model (blockHMM), to infer switches in discrete latent states that govern the choice sequences across blocks of trials. Our results revealed a remarkable degree of mixing between different strategies even in expert animals, such that model-free and inference-based learning modes often co-existed within single sessions. Together, these results invite a re-evaluation of the stationarity of behavior during dynamic foraging, provide a comprehensive set of tools to characterize the evolution of learning strategies, and form the basis of understanding neural circuits involved in different modes of behavior within this domain.