Biology subjects

Tomov, M. S.

Publications and source records attributed to Tomov, M. S..

4 recordsLinked to original sources

bioRxiv2024

Neural Prioritisation of Past Solutions Supports Generalisation

Generalisation from past experience is an important feature of intelligent systems. When faced with a new task, one efficient computational approach is to evaluate solutions to earlier tasks as candidates for reuse. Consistent with this idea, we found that human participants (n=38) learned optimal solutions to a set of training tasks and generalised them to novel test tasks in a reward selective manner. This behaviour was consistent with a computational process based on the successor representation known as successor features and generalised policy improvement (SF&GPI). Neither model-free perseveration or model-based control using a complete model of the environment could explain choice behaviour. Decoding from functional magnetic resonance imaging data revealed that solutions from the SF&GPI algorithm were activated on test tasks in visual and prefrontal cortex. This activation had a functional connection to behaviour in that stronger activation of SF&GPI solutions in visual areas was associated with increased behavioural reuse. These findings point to a possible neural implementation of an adaptive algorithm for generalisation across tasks.

neuroscience↗

bioRxiv2023

Competitive integration of time and reward explains value-sensitive foraging decisions and frontal cortex ramping dynamics

Patch foraging presents a ubiquitous decision-making process in which animals decide when to abandon a resource patch of diminishing value to pursue an alternative. We developed a virtual foraging task in which mouse behavior varied systematically with patch value. Mouse behavior could be explained by a model integrating time and rewards antagonistically, scaled by a latent patience state. The model accounted for deviations from predictions of optimal foraging theory. Neural recordings throughout frontal areas revealed encoding of decision variables from the integrator model, most robustly in frontal cortex. Regression modeling followed by unsupervised clustering identified a subset of ramping neurons. These neurons firing rates ramped up gradually (up to tens of seconds), were inhibited by rewards, and were better described as a continuous ramp than a discrete stepping process. Together, these results identify integration via frontal cortex ramping dynamics as a candidate mechanism for solving patch foraging problems.

neuroscience↗

bioRxiv2022

The Neural Architecture of Theory-based Reinforcement Learning

Humans learn internal models of the environment that support efficient planning and flexible generalization in complex, real-world domains. Yet it remains unclear how such internal models are represented and learned in the brain. We approach this question within the framework of theory-based reinforcement learning, a strong form of model-based reinforcement learning in which the model is an intuitive theory - a rich, abstract, causal model of the environment built on a natural ontology of physical objects, intentional agents, relations, and goals. We used a theory-based reinforcement learning model to analyze brain data from human participants learning to play different Atari-style video games while undergoing functional MRI. Theories inferred by the theory-based model explained the signal in inferior frontal gyrus and other prefrontal areas better than several alternative models. Brain activity increased in response to theory update events in inferior frontal gyrus, occipital cortex, and fusiform gyrus, with separate learning signals for different theory components. This corresponded with a transient strengthening of theory representations in those regions. Finally, the effective connectivity pattern during theory updating suggests that information flows top-down from theory-coding regions in the prefrontal cortex to theory updating regions in occipital and temporal cortex. These results are consistent with a neural architecture in which top-down theory representations originating in prefrontal regions shape sensory predictions in visual areas, where factorized theory prediction errors are computed and in turn trigger bottom-up updates of the theory. This initial sketch provides a foundation for understanding of the neural representations and computations that support efficient theory-based reinforcement learning in complex, naturalistic environments.

neuroscience↗

bioRxiv2020

Causal inference gates corticostriatal learning

Attributing outcomes to your own actions or to external causes is essential for appropriately learning which actions lead to reward and which actions do not. Our previous work showed that this type of credit assignment is best explained by a Bayesian reinforcement learning model which posits that beliefs about the causal structure of the environment modulate reward prediction errors (RPEs) during action value updating. In this study, we investigated the neural circuits underlying reinforcement learning that are influenced by causal beliefs using functional magnetic resonance imaging (fMRI) while human participants (N = 31; 13 males, 18 females) completed a behavioral task that manipulated beliefs about causal structure. We found evidence that RPEs modulated by causal beliefs are represented in posterior putamen, while standard (unmodulated) RPEs are represented in ventral striatum. Further analyses revealed that beliefs about causal structure are represented in anterior insula and inferior frontal gyrus. Finally, structural equation modeling revealed effective connectivity from anterior insula to posterior putamen. Together, these results are consistent with a neural architecture in which causal beliefs in anterior insula are integrated with prediction error signals in posterior putamen to update action values. Significance StatementLearning which actions lead to reward - a process known as reinforcement learning - is essential for survival. Inferring the causes of observed outcomes - a process known as causal inference - is crucial for appropriately assigning credit to ones own actions and restricting learning to effective action-outcome contingencies. Previous studies have linked reinforcement learning to the striatum and causal inference to prefrontal regions, yet how these neural processes interact to guide adaptive behavior remains poorly understood. Here, we found evidence that causal beliefs represented in the prefrontal cortex modulate action value updating in posterior striatum, separately from the unmodulated action value update in ventral striatum posited by standard reinforcement learning models.

neuroscience↗