Biology subjects

Niv, Y.

Publications and source records attributed to Niv, Y..

11 recordsLinked to original sources

bioRxiv2020

Orbitofrontal cortex and learning predictions of state transitions

Learning the transition structure of the environment - the probabilities of transitioning from one environmental state to another - is a key prerequisite for goal-directed planning and model-based decision making. To investigate the role of the orbitofrontal cortex (OFC) in goal-directed planning and decision making, we used fMRI to assess univariate and multivariate activity in the OFC while humans experienced state transitions that varied in degree of surprise. Converging with recent evidence, we found that OFC activity was related to greater learning about transition structure. However, the observed relationship was inconsistent with a straightforward interpretation of OFC activity as representing a state prediction error that would facilitate learning of transitions via error-correcting mechanisms. The state prediction error hypothesis predicts that OFC activity at the time of observing an outcome should increase expectation of that same observed outcome on subsequent trials. Instead, our results showed that OFC activity was associated with increased expectation of the more probable outcome; that is, with more optimal predictions. These results suggest that the OFC is involved in updating or reinforcing a learned transition model on a trial-by-trial basis, specifically for the currently observed cue-outcome associations. Our findings add to the evidence of OFC involvement in learning state-to-state transition structure, while providing new constraints for algorithmic hypotheses regarding how these transitions are learned. Significance StatementThe orbitofrontal cortex (OFC) has been implicated in model-based decision making--the kind of decisions that result from planning using an "environment model" of how current actions affect our future states. However, the widely suggested role of the OFC in representing expected values of future states is not sufficient to explain why the OFC would be critical for planning in particular. A new line of evidence implicates the OFC in learning about transition structure of the environment - a key component of the "environment model" used for planning. We investigate this function, adding to the growing literature on the role of the OFC in learning and decision making, while unveiling new questions about the algorithmic role of OFC in goal-directed planning.

neuroscience↗

bioRxiv2019

Modeling the Hemodynamic Response Function for Prediction Errors in the Ventral Striatum

To make sensible inferences about neural activation from fMRI data it is important to accurately model the hemodynamic response function (HRF), i.e., the hemodynamic response evoked by a punctuate neural event. HRF models have been derived for sensory areas, where it is relatively clear what events cause a neural impulse response. However, this is obviously harder to do for higher order cortices such as prefrontal areas. Therefore, one HRF model is commonly used for analyzing activity throughout the brain, despite the fact that hemodynamics are known to vary across regions. For instance, several fMRI studies use a canonical HRF to analyze ventral striatum (VS) activity where converging evidence indicates that reward prediction error signals drive neural activity. However, the VS is a target of prominent dopaminergic projections, known to modulate vasculature and affect BOLD activity, suggesting that the HRF in the VS may be especially different from those in other brain areas. To address this, we use data from an experiment focused on learning from prediction-error signals to derive a VS-specific HRF model (VS-HRF). We show that this new VS-HRF increases statistical power in model comparison. Our result is of particular relevance to studies comparing computational models of learning and/or decision making in the VS, and for connectivity analyses, where the use of an (even slightly) inaccurate HRF model can lead to erroneous conclusions. More broadly, our study highlights the importance of the choice of HRF model in determining the significance of the results obtained in classical univariate fMRI analysis.

neuroscience↗

bioRxiv2019

Joy division: Reward prediction errors create event boundaries in memory

We remember when things change. Particularly salient are experiences where there is a change in rewards, eliciting reward prediction errors (RPEs). This feature of memory may be useful because it can help us find greater rewards and avoid lesser ones in the future. How do RPEs influence our memory of those experiences? One idea is that this signal directly enhances the encoding of memory. Another, not mutually exclusive, idea is that the RPE signals a deeper change in the environment, and leads to the mnemonic separation of subsequent experiences from what came before, thereby creating a new latent context and a more separate memory trace. We tested this in four experiments in which participants learned to predict rewards associated with a series of images within visually-distinct \"rooms.\" High magnitude RPEs indicated a change in the underlying distribution of rewards. To test whether these large RPEs created a new latent context, we first assessed recognition priming for sequential pairs that contained or did not contain a high-RPE event, as well as out-of-sequence pairs (Exp. 1: n=27 & Exp. 2: n=83). We found evidence of recognition priming for both sequential pair types, including the pair with the high-RPE event, indicating that the high-RPE event is bound to its predecessor in memory. Given that high-RPE events are themselves preferentially remembered (Rouhani et al, 2018), we next tested recognition priming for pairs that had one item in between them (i.e. the pairs were either across a high-RPE event or not), where none of the tested items were high-RPE items (Exp. 3: n=85). Here, sequential pairs across a high-RPE no longer showed recognition priming whereas pairs within the same latent reward state did, providing initial evidence for an RPE-modulated event boundary. We then investigated whether RPE event boundaries disrupt temporal memory of those events (Exp. 4). After reward learning, we asked participants to order and estimate the distance between two events that had either included a high-RPE event between them, or not. We found (n=49) and replicated (n=77) worse sequence memory for events across a high-RPE event. Altogether, these findings demonstrate that high-RPE events are both more strongly encoded and act as event boundaries that interrupt the sequential integration of events. We captured these effects in a variant of the Context Maintenance and Retrieval model (CMR; Polyn, Norman & Kahana, 2009), modified to incorporate RPEs into the encoding process.

neuroscience↗

bioRxiv2019

Dopamine transients delivered in learning contexts do not act as model-free prediction errors

Dopamine neurons fire transiently in response to unexpected rewards. These neural correlates are proposed to signal the reward prediction error described in model-free reinforcement learning algorithms. This error term represents the unpredicted or excess value of the rewarding event. In model-free reinforcement learning, this value is then stored as part of the learned value of any antecedent cues, contexts or events, making them intrinsically valuable, independent of the specific rewarding event that caused the prediction error. In support of equivalence between dopamine transients and this model-free error term, proponents cite causal optogenetic studies showing that artificially induced dopamine transients cause lasting changes in behavior. Yet none of these studies directly demonstrate the presence of cached value under conditions appropriate for associative learning. To address this gap in our knowledge, we conducted three studies where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquired value and instead entered into value-independent associative relationships with the other cues or rewards. These results show that dopamine transients, constrained within appropriate learning situations, support valueless associative learning.

neuroscience↗

bioRxiv2018

Rat orbitofrontal ensemble activity contains a multiplexed but value-invariant representation of task structure in an odor sequence task

The orbitofrontal cortex (OFC) has long been implicated in signaling information about expected outcomes to facilitate adaptive or flexible behavior. Current proposals focus on signaling of expected reward values versus the representation of a value-agnostic cognitive map of the task. While often suggested as mutually exclusive, these alternatives may represent two extreme ends of a continuum determined by the complexity of the environment and the subjects experience in it. As learning proceeds, an initial, detailed cognitive map might be acquired, based largely on external information. With more experience, this hypothesized map can then be tailored to include relevant abstract hidden cognitive constructs. This might default to expected values in situations where other attributes are minimized or largely irrelevant, whereas in richer tasks, a more detailed structure might continue to be represented, at least where relevant to behavior, and possibly alongside value. Here we sought to arbitrate between these options by recording single unit activity from the OFC in rats navigating an odor sequence task analogous to a spatial maze. The odor sequences provided a clearly mappable state space, with 24 unique \"positions\" defined by sensory information, likelihood of reward, or both. Consistent with the hypothesis that the OFC represents a cognitive map tailored to the subjects intentions or plans, we found a close correspondence between how subjects behavior suggested they were using the sequences, and the neural representations of the sequences in OFC ensembles. Multiplexed with this value-invariant representation of the task, we also found a representation of the expected value at each location. Thus value and task structure are co-existing and potentially dissociable components of the neural code in OFC.

neuroscience↗

bioRxiv2018

Neural Signatures of Prediction Errors in a Decision-making Task are Modulated by Action Execution Failures

Decisions must be implemented through actions, and actions are prone to error. As such, when an expected outcome is not obtained, an individual should not only be sensitive to whether the choice itself was suboptimal, but also whether the action required to indicate that choice was executed successfully. The intelligent assignment of credit to action execution versus action selection has clear ecological utility for the learner. To explore this scenario, we used a modified version of a classic reinforcement learning task in which feedback indicated if negative prediction errors were, or were not, associated with execution errors. Using fMRI, we asked if prediction error computations in the human striatum, a key substrate in reinforcement learning and decision making, are modulated when a failure in action execution results in the negative outcome. Participants were more tolerant of non-rewarded outcomes when these resulted from execution errors versus when execution was successful but the reward was withheld. Consistent with this behavior, a model-driven analysis of neural activity revealed an attenuation of the signal associated with negative reward prediction error in the striatum following execution failures. These results converge with other lines of evidence suggesting that prediction errors in the mesostriatal dopamine system integrate high-level information during the evaluation of instantaneous reward outcomes.

neuroscience↗

bioRxiv2018

Representational structure or task structure? Bias in neural representational similarity analysis and a Bayesian method for reducing bias

The activity of neural populations in the brains of humans and animals can exhibit vastly different spatial patterns when faced with different tasks or environmental stimuli. The degree of similarity between these neural activity patterns in response to different events is used to characterize the representational structure of cognitive states in a neural population. The dominant methods of investigating this similarity structure first estimate neural activity patterns from noisy neural imaging data using linear regression, and then examine the similarity between the estimated patterns. Here, we show that this approach introduces spurious bias structure in the resulting similarity matrix, in particular when applied to fMRI data. This problem is especially severe when the signal-to-noise ratio is low and in cases where experimental conditions cannot be fully randomized in a task. We propose Bayesian Representational Similarity Analysis (BRSA), an alternative method for computing representational similarity, in which we treat the covariance structure of neural activity patterns as a hyper-parameter in a generative model of the neural data. By marginalizing over the unknown activity patterns, we can directly estimate this covariance structure from imaging data. This method offers significant reductions in bias and allows estimation of neural representational similarity with previously unattained levels of precision at low signal-to-noise ratio. The probabilistic framework allows for jointly analyzing data from a group of participants. The method can also simultaneously estimate a signal-to-noise ratio map that shows where the learned representational structure is supported more strongly. Both this map and the learned covariance matrix can be used as a structured prior for maximum a posteriori estimation of neural activity patterns, which can be further used for fMRI decoding. We make our tool freely available in Brain Imaging Analysis Kit (BrainIAK).\n\nAuthor summaryWe show the severity of the bias introduced when performing representational similarity analysis (RSA) based on neural activity pattern estimated within imaging runs. Our Bayesian RSA method significantly reduces the bias and can learn a shared representational structure across multiple participants. We also demonstrate its extension as a new multi-class decoding tool.

neuroscience↗

bioRxiv2018

Sequential replay of non-spatial task states in the human hippocampus

Neurophysiological research has found that previously experienced sequences of spatial events are reactivated in the hippocampus of rodents during wakeful rest. This phenomenon has become a cornerstone of modern theories of memory and decision making. Yet, whether hippocampal sequence reactivation at rest is of general importance also for humans and non-spatial tasks has remained unclear. Here, we investigated sequences of fMRI BOLD activation patterns in humans during wakeful rest following a sequential non-spatial decision-making task. We found that pattern reactivations within the human hippocampus reflected the order of previous task state sequences, and that the extent of this offline reactivation was related to the on-task representation of task states in the orbitofrontal cortex. Permutation analyses and fMRI signal simulations confirmed that these results reflected underlying BOLD activity, and showed that our novel statistical analyses are, in principle, sensitive to sequential neural events occurring as fast as one hundred milliseconds apart. Our results support the importance of sequential reactivation in the human hippocampus for decision making, and establish the feasibility of investigating such rapid signals with fMRI, despite its substantial temporal limitations.\n\nHighlightsO_LIWe provide fMRI evidence for sequential pattern reactivation in the human hippocampus\nC_LIO_LISequences of patterns reflect states from a sequential, non-spatial decision-making task\nC_LIO_LISimulations show that our novel fMRI analysis is sensitive to fast sequences of sub-second neural events\nC_LIO_LIResults support the importance of sequential reactivation in the human hippocampus for decision making\nC_LI

neuroscience↗

bioRxiv2018

A pupillary index of susceptibility to decision biases

Under what conditions do humans systematically deviate from rational decision making? Here we show that pupillary indices of low neural gain are associated with strong and consistent biases across six different extensively-studied decision making tasks, whereas indices of high gain are associated with weak or absent biases. Lower susceptibility to biases, however, comes at the cost of indecisiveness, or alternatively, prolonged deliberation time. We explain the association between low gain and strong biases as reflecting a broader information integration process that gives greater weight to weak biasing influences. The findings underscore the role of pupil-linked brain states in the generation of decision making biases.\n\nSignificance\"Framing effects\" are demonstrations that peoples decisions can be biased by the way a decision problem is presented, and consequently, people can make decisions that violate the principles of rationality. Using a set of classic decision-making tasks, we show that pupil dilation, previously linked to levels of the neuromodulator norepinephrine and to a tradeoff between narrowly focused and broadly integrative modes of information processing, distinguishes between people who are consistently biased and people who are relatively immune to these effects. Our findings suggest that norepinephrine may drive these individual differences, and that a narrowly focused mode of information processing confers relative immunity to these decision making biases, whereas the integration of a wider range of information results in greater susceptibility.

neuroscience↗

bioRxiv2017

A state representation for reinforcement learning and decision-making in the orbitofrontal cortex

Despite decades of research, the exact ways in which the orbitofrontal cortex (OFC) influences cognitive function have remained mysterious. Anatomically, the OFC is characterized by remarkably broad connectivity to sensory, limbic and subcortical areas, and functional studies have implicated the OFC in a plethora of functions ranging from facial processing to value-guided choice. Notwithstanding such diversity of findings, much research suggests that one important function of the OFC is to support decision making and reinforcement learning. Here, we describe a novel theory that posits that OFCs specific role in decision-making is to provide an up-to-date representation of task-related information, called a state representation. This representation reflects a mapping between distinct task states and sensory as well as unobservable information. We summarize evidence supporting the existence of such state representations in rodent and human OFC and argue that forming these state representations provides a crucial scaffold that allows animals to efficiently perform decision making and reinforcement learning in high-dimensional and partially observable environments. Finally, we argue that our theory offers an integrating framework for linking the diversity of functions ascribed to OFC and is in line with its wide ranging connectivity.

neuroscience↗

bioRxiv2017

Dissociable effects of surprising rewards on learning and memory

The extent to which rewards deviate from learned expectations is tracked by a signal known as a \"reward prediction error\", but it is unclear how this signal interacts with episodic memory. Here, we investigated whether learning in a high-risk environment, with frequent large prediction errors, gives rise to higher fidelity memory traces than learning in a low-risk environment. In Experiment 1, we showed that higher magnitude prediction errors, positive or negative, improved recognition memory for trial-unique items. Participants also increased their learning rate after large prediction errors. In addition, there was an overall higher learning rate in the low-risk environment. Although unsigned prediction errors enhanced memory and increased learning rate, we did not find a relationship between learning rate and memory, suggesting that these two effects were due to separate underlying mechanisms. In Experiment 2, we replicated these results with a longer task that posed stronger memory demands and allowed for more learning. We also showed improved source and sequence memory for high-risk items. In Experiment 3, we controlled for the difficulty of learning in the two risk environments, again replicating the previous results. Moreover, equating the range of prediction errors in the two risk environments revealed that learning in a high-risk context enhanced episodic memory above and beyond the effect of prediction errors to individual items. In summary, our results across three studies showed that (absolute) prediction error magnitude boosted both episodic memory and incremental learning, but the two effects were not correlated, suggesting distinct underlying systems.

animal behavior and cognition↗