Biology subjects

Schoenbaum, G.

Publications and source records attributed to Schoenbaum, G..

29 records · Page 2Linked to original sources

bioRxiv2021

Validating the Representational Space of DeepReinforcement Learning Models of Behavior withNeural Data

Deep Reinforcement Learning (Deep RL) agents have in recent years emerged as successful models of animal behavior in a variety of complex learning tasks, as exemplified by Song et al. [2017]. As agents are typically trained to mimic an animal subject, the emphasis in past studies on behavior as a means of evaluating the fitness of models to experimental data is only natural. But the true power of Deep RL agents lies in their ability to learn neural computations and codes that generate a particular behavior--factors that are also of great relevance and interest to computational neuroscience. On that basis, we believe that model evaluation should include an examination of neural representations and validation against neural recordings from animal subjects. In this paper, we introduce a procedure to test hypotheses about the relationship between internal representations of Deep RL agents and those in animal neural recordings. Taking a sequential learning task as a running example, we apply our method and show that the geometry of representations learnt by artificial agents is similar to that of the biological subjects, and that such similarities are driven by shared information in some latent space. Our method is applicable to any Deep RL agent that learns a Markov Decision Process, and as such enables researchers to assess the suitability of more advanced Deep Learning modules, or map hierarchies of representations to different parts of a circuit in the brain, and help shed light on their function. To demonstrate that point, we conduct an ablation study to deduce that, in the sequential task under consideration, temporal information plays a key role in molding a correct representation of the task.

neuroscience↗

bioRxiv2020

Orbitofrontal cortex and learning predictions of state transitions

Learning the transition structure of the environment - the probabilities of transitioning from one environmental state to another - is a key prerequisite for goal-directed planning and model-based decision making. To investigate the role of the orbitofrontal cortex (OFC) in goal-directed planning and decision making, we used fMRI to assess univariate and multivariate activity in the OFC while humans experienced state transitions that varied in degree of surprise. Converging with recent evidence, we found that OFC activity was related to greater learning about transition structure. However, the observed relationship was inconsistent with a straightforward interpretation of OFC activity as representing a state prediction error that would facilitate learning of transitions via error-correcting mechanisms. The state prediction error hypothesis predicts that OFC activity at the time of observing an outcome should increase expectation of that same observed outcome on subsequent trials. Instead, our results showed that OFC activity was associated with increased expectation of the more probable outcome; that is, with more optimal predictions. These results suggest that the OFC is involved in updating or reinforcing a learned transition model on a trial-by-trial basis, specifically for the currently observed cue-outcome associations. Our findings add to the evidence of OFC involvement in learning state-to-state transition structure, while providing new constraints for algorithmic hypotheses regarding how these transitions are learned. Significance StatementThe orbitofrontal cortex (OFC) has been implicated in model-based decision making--the kind of decisions that result from planning using an "environment model" of how current actions affect our future states. However, the widely suggested role of the OFC in representing expected values of future states is not sufficient to explain why the OFC would be critical for planning in particular. A new line of evidence implicates the OFC in learning about transition structure of the environment - a key component of the "environment model" used for planning. We investigate this function, adding to the growing literature on the role of the OFC in learning and decision making, while unveiling new questions about the algorithmic role of OFC in goal-directed planning.

neuroscience↗

bioRxiv2020

Prospective representations in rat orbitofrontal ensembles

The orbitofrontal cortex (OFC) has been proposed to encode expected outcomes, which is thought to be important for outcome-directed behavior. However, such neural encoding can also often be explained by the recall of information about the recent past. To dissociate the retrospective and prospective aspects of encoding in the OFC, we designed a non-spatial, continuous, alternating odor-sequence task that mimicked a continuous T-maze. The task consisted of two alternating sequences of four odor-guided trials (2 sequences x 4 positions). In each trial, rats were asked to make a "go" or "no-go" action based on a fixed odor-reward contingency. Odors at both the first and last positions were distinct across the two sequences, such that they resembled unique paths in the past and future, respectively; odors at positions in between were the same and thus resembled a common path. We trained classifiers using neural activity to distinguish between either sequences or positions and asked whether the neural activity patterns in the common path were more like the ones in the past or the future. We found a proximal prospective code for sequence information as well as a distal prospective code for positional information, the latter of which was closely associated with rats ability to predict future outcomes. This study demonstrates a prospective behaviorally-relevant predictive code in rat OFC.

neuroscience↗

bioRxiv2020

Devaluation-sensitive responding to preconditioned cues requires orbitofrontal cortex during initial cue-cue learning

The orbitofrontal cortex (OFC) is necessary for value inference in tests of model-based reasoning. This ability could be accounted for by either representation of value or by representation of broader associative structure. Our lab recently reported correlates of both value and of valueless associative structure in OFC using single-unit recording (Sadacca et al., 2018). This incidental stimulus-stimulus representation was surprising since OFC was thought to be involved only when items of biological significance were driving responses. However, we did not assess whether this activity was necessary for encoding the associative information that would contribute to value inference during probe testing. Here, we used optogenetic OFC inhibition during sensory preconditioning to test this. We found that OFC inhibition during preconditioning impaired value inference during the probe test, demonstrating that the correlates we previously observed are not simply downstream readouts of sensory processing and instead contribute to encoding valueless sensory associative information.

neuroscience↗

bioRxiv2020

Human orbitofrontal cortex is necessary for behavior based on inferred, not experienced outcomes

When direct experience is unavailable, animals and humans can imagine or infer the future to guide decisions. Behavior based on direct experience versus inference may recruit distinct but overlapping brain circuits. In rodents, the orbitofrontal cortex (OFC) contains neural signatures of inferred outcomes, and OFC is necessary for behavior that requires inference but not for responding driven by direct experience. In humans, OFC activity is also correlated with inferred outcomes, but it is unclear whether OFC activity is required for inference-based behavior. To test this, we used non-invasive network-based continuous theta burst stimulation (cTBS) to target lateral OFC networks in the context of a sensory preconditioning task that was designed to isolate inference-based behavior from responding that can be based on direct experience alone. We show that relative to sham, cTBS targeting this network impairs reward-related behavior in conditions in which outcome expectations have to be mentally inferred. In contrast, OFC-targeted stimulation does not impair behavior that can be based on previously experienced stimulus-outcome associations. These findings suggest that activity in the targeted OFC network supports decision making when outcomes have to be mentally simulated, providing converging cross-species evidence for a critical role of OFC in model-based but not model-free control of behavior.

neuroscience↗

bioRxiv2019

Targeted stimulation of human orbitofrontal networks disrupts outcome-guided behavior

Outcome-guided behavior requires knowledge about the current value of expected outcomes. Such behavior can be isolated in the reinforcer devaluation task, which assesses the ability to infer the current value of rewards after devaluation. Animal lesion studies demonstrate that orbitofrontal cortex (OFC) is necessary for normal behavior in this task, but a causal role for human OFC in outcome-guided behavior has not been established. Here we used sham-controlled non-invasive continuous theta-burst stimulation (cTBS) to temporarily disrupt human OFC network activity prior to devaluation of food odor rewards in a between-subjects design. Subjects in the sham group appropriately avoided Pavlovian cues associated with devalued food odors. However, subjects in the stimulation group persistently chose those cues, even though devaluation of food odors themselves was unaffected by cTBS. This behavioral impairment was mirrored in changes in resting-stated functional magnetic resonance imaging (rs-fMRI) activity, such that subjects in the stimulation group exhibited reduced global OFC network connectivity after cTBS, and the magnitude of this reduction was correlated with choices after devaluation. These findings demonstrate the feasibility of indirectly targeting the human OFC with non-invasive cTBS, and indicate that OFC is specifically required for inferring the value of expected outcomes.

neuroscience↗

bioRxiv2019

Dopamine neuron ensembles signal the content of sensory prediction errors

Dopamine neurons respond to errors in predicting value-neutral sensory information. These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in individual neuron or average neural activity. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event. Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.

neuroscience↗

bioRxiv2019

Dopamine transients delivered in learning contexts do not act as model-free prediction errors

Dopamine neurons fire transiently in response to unexpected rewards. These neural correlates are proposed to signal the reward prediction error described in model-free reinforcement learning algorithms. This error term represents the unpredicted or excess value of the rewarding event. In model-free reinforcement learning, this value is then stored as part of the learned value of any antecedent cues, contexts or events, making them intrinsically valuable, independent of the specific rewarding event that caused the prediction error. In support of equivalence between dopamine transients and this model-free error term, proponents cite causal optogenetic studies showing that artificially induced dopamine transients cause lasting changes in behavior. Yet none of these studies directly demonstrate the presence of cached value under conditions appropriate for associative learning. To address this gap in our knowledge, we conducted three studies where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquired value and instead entered into value-independent associative relationships with the other cues or rewards. These results show that dopamine transients, constrained within appropriate learning situations, support valueless associative learning.

neuroscience↗

bioRxiv2019

Causal evidence supporting the proposal that dopamine transients function as a temporal difference prediction error

Reward-evoked dopamine is well-established as a prediction error. However the central tenet of temporal difference accounts - that similar transients evoked by reward-predictive cues also function as errors - remains untested. To address this, we used two phenomena, second-order conditioning and blocking, in order to examine the role of dopamine in prediction error versus reward prediction. We show that optogenetically-shunting dopamine activity at the start of a reward-predicting cue prevents second-order conditioning without affecting blocking. These results support temporal difference accounts by providing causal evidence that cue-evoked dopamine transients function as prediction errors.

neuroscience↗

bioRxiv2018

Rat orbitofrontal ensemble activity contains a multiplexed but value-invariant representation of task structure in an odor sequence task

The orbitofrontal cortex (OFC) has long been implicated in signaling information about expected outcomes to facilitate adaptive or flexible behavior. Current proposals focus on signaling of expected reward values versus the representation of a value-agnostic cognitive map of the task. While often suggested as mutually exclusive, these alternatives may represent two extreme ends of a continuum determined by the complexity of the environment and the subjects experience in it. As learning proceeds, an initial, detailed cognitive map might be acquired, based largely on external information. With more experience, this hypothesized map can then be tailored to include relevant abstract hidden cognitive constructs. This might default to expected values in situations where other attributes are minimized or largely irrelevant, whereas in richer tasks, a more detailed structure might continue to be represented, at least where relevant to behavior, and possibly alongside value. Here we sought to arbitrate between these options by recording single unit activity from the OFC in rats navigating an odor sequence task analogous to a spatial maze. The odor sequences provided a clearly mappable state space, with 24 unique \"positions\" defined by sensory information, likelihood of reward, or both. Consistent with the hypothesis that the OFC represents a cognitive map tailored to the subjects intentions or plans, we found a close correspondence between how subjects behavior suggested they were using the sequences, and the neural representations of the sequences in OFC ensembles. Multiplexed with this value-invariant representation of the task, we also found a representation of the expected value at each location. Thus value and task structure are co-existing and potentially dissociable components of the neural code in OFC.

neuroscience↗

bioRxiv2017

Rethinking dopamine prediction errors

Midbrain dopamine neurons are commonly thought to report a reward prediction error, as hypothesized by reinforcement learning theory. While this theory has been highly successful, several lines of evidence suggest that dopamine activity also encodes sensory prediction errors unrelated to reward. Here we develop a new theory of dopamine function that embraces a broader conceptualization of prediction errors. By signaling errors in both sensory and reward predictions, dopamine supports a form of reinforcement learning that lies between model-based and model-free algorithms. This account remains consistent with current canon regarding the correspondence between dopamine transients and reward prediction errors, while also accounting for new data suggesting a role for these signals in phenomena such as sensory preconditioning and identity unblocking, which ostensibly draw upon knowledge beyond reward predictions.

neuroscience↗