Biology subjects

Tomov, M.

Publications and source records attributed to Tomov, M..

4 recordsLinked to original sources

bioRxiv2019

Multi-Task Reinforcement Learning in Humans

The ability to transfer knowledge across tasks and generalize to novel ones is an important hallmark of human intelligence. Yet not much is known about human multi-task reinforcement learning. We study participants behavior in a novel two-step decision making task with multiple features and changing reward functions. We compare their behavior to two state-of-the-art algorithms for multi-task reinforcement learning, one that maps previous policies and encountered features to new reward functions and one that approximates value functions across tasks, as well as to standard model-based and model-free algorithms. Across three exploratory experiments and a large preregistered experiment, our results provide strong evidence for a strategy that maps previously learned policies to novel scenarios. These results enrich our understanding of human reinforcement learning in complex environments with changing task demands.

neuroscience↗

bioRxiv2018

Discovery of Hierarchical Representations for Efficient Planning

We propose that humans spontaneously organize environments into clusters of states that support hierarchical planning, enabling them to tackle challenging problems by breaking them down into sub-problems at various levels of abstraction. People constantly rely on such hierarchical presentations to accomplish tasks big and small - from planning ones day, to organizing a wedding, to getting a PhD - often succeeding on the very first attempt. We formalize a Bayesian model of hierarchy discovery that explains how humans discover such useful abstractions. Building on principles developed in the robotics literature, the model predicts that hierarchy discovery should be sensitive to the topological structure, reward distribution, and distribution of tasks in the environment. In five simulations, we show that the model accounts for previously reported effects of environment structure on planning behavior, such as detection of bottleneck states and transitions. We then test the novel predictions of the model in seven behavioral experiments, demonstrating how the distribution of tasks and rewards can additionally influence planning behavior via the discovered hierarchy, sometimes facilitating and sometimes hindering performance. We also show that people use uncertainty to guide their learning in a way that is informative for hierarchy discovery. Finally, we propose how hierarchy discovery and hierarchical planning might be implemented in the brain. Together, these findings present an important advance in our understanding of how the brain might use Bayesian inference to uncover and exploit the hidden hierarchical structure of the world.

neuroscience↗

bioRxiv2018

Dissociable Neural Correlates of Uncertainty Underlie Different Exploration Strategies

Most real-world decisions involve a delicate balance between exploring unfamiliar alternatives and committing to the best known option. Uncertainty lies at the core of this "explore-exploit" dilemma, for if all options were perfectly known, there would be no need to explore. Yet despite the prominent role of uncertainty-guided exploration in decision making, evidence for its neural implementation is still sparse. We investigated this question with model-based fMRI (n = 31) using a two-armed bandit task that independently manipulates two forms of uncertainty underlying different exploration strategies. The relative uncertainty between the two options was correlated with BOLD activity in right rostrolateral prefrontal cortex and drove directed exploration, a strategy that adds an uncertainty bonus to each option. The total uncertainty across the two options was correlated with activity in left dorsolateral prefrontal cortex and drove random exploration, a strategy that increases choice stochasticity in proportion to total uncertainty. The subjective estimates of uncertainty from both regions were predictive of subject choices. These results are consistent with a hybrid computational architecture in which different uncertainty computations are performed separately and then combined by downstream decision circuits to compute choice.

neuroscience↗

bioRxiv2017

Neural Computations Underlying Causal Structure Learning

Behavioral evidence suggests that beliefs about causal structure constrain associative learning, determining which stimuli can enter into association, as well as the functional form of that association. Bayesian learning theory provides one mechanism by which structural beliefs can be acquired from experience, but the neural basis of this mechanism is unknown. A recent study (Gershman, 2017) proposed a unified account of the elusive role of \"context\" in animal learning based on Bayesian updating of beliefs about the structure of causal relationships between contexts and cues in the environment. The model predicts that the computations which arbitrate between these abstract causal structures are distinct from the computations which learn the associations between particular stimuli under a given structure. In this study, we used fMRI with male and female human subjects to interrogate the neural correlates of these two distinct forms of learning. We show that structure learning signals are encoded in rostrolateral prefrontal cortex and the angular gyrus, anatomically distinct from correlates of associative learning. Within-subject variability in the encoding of these learning signals predicted variability in behavioral performance. Moreover, representational similarity analysis suggests that some regions involved in both forms of learning, such as parts of the inferior frontal gyrus, may also encode the full probability distribution over causal structures. These results provide evidence for a neural architecture in which structure learning guides the formation of associations.\n\nSignificance StatementAnimals are able to infer the hidden structure behind causal relations between stimuli in the environment, allowing them to generalize this knowledge to stimuli they have never experienced before. A recently published computational model based on this idea provided a parsimonious account of a wide range of phenomena reported in the animal learning literature, suggesting that the neural mechanisms dedicated to learning this hidden structure are distinct from those dedicated to acquiring particular associations between stimuli. Here we validate this model by measuring brain activity during a task which dissociates structure learning from associative learning. We show that different brain networks underlie the two forms of learning and that the neural signal corresponding to structure learning predicts future behavioral performance.

neuroscience↗