Biology subjects

Palminteri, S.

Publications and source records attributed to Palminteri, S..

At least 19 recordsLinked to original sources

bioRxiv2025

Dissociating the functional role of the para-hippocampal and the parietal cortex in human multi-step reinforcement learning

Many real-life decisions involve a tension between short-term and long-term outcomes, which requires forward-looking abilities. In reinforcement learning, this tension arises at the initial stage of multi-step learning and decision tasks, where forward-looking decisions collect smaller immediate rewards but govern the transition to more advantageous second-stage states. Here, we investigated the neural mechanisms underlying such forward-looking decisions in a cohort of healthy participants undergoing fMRI scanning (N=28). Behavioral results confirmed that participants were able to learn concurrently reward values and state-transition probabilities. By contrasting BOLD signal elicited by first-stage versus second-stage at the time of decision, we isolated a brain network, with central nodes in the bilateral parahippocampal cortex (BA37), whose activity correlate with forward-looking choices. BOLD activity in another network, including the bilateral parietal cortex, correlated with structure learning signal at the time of outcome processing. Our results shed new light on the neural bases of model-based reinforcement learning by suggesting a specific role of the parahippocampal cortex in forward planning and the parietal cortex in learning the Markovian structure of the task.

neuroscience↗

bioRxiv2025

Reference Point-Dependent Reinforcement Learning in Humans and Rats

Previous studies indicate that rewards and punishments in reinforcement learning are encoded in a relative manner. Reference point-dependence, a valuation bias shared by eminent adaptation level and prospect theories, is often cited as the underlying computational mechanism. However, whether these behavioural and computational mechanisms are preserved across species is unknown. To fill this gap, we designed a reinforcement learning task that was adapted for both humans (homo sapiens sapiens) and rats (rattus norvegicus). Behavioural analyses indicated robust relative value encoding in both species. At the computational level, reference point-dependence provided a reliable account of reinforcement learning behaviour in humans and rats. Despite these major similarities between species, some distinct differences in behavioural and computational modelling parameters were also observed. Overall, our study demonstrates that relative value encoding is a robust feature of reinforcement learning that is conserved across human and rat species.

molecular biology↗

bioRxiv2023

Neural and computational underpinnings of biased confidence in human reinforcement learning

While navigating a fundamentally uncertain world, humans and animals constantly produce subjective confidence judgments, thereby evaluating the probability of their decisions, actions or statements being correct. Confidence typically correlates with neural activity positively in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.

animal behavior and cognition↗

bioRxiv2022

The role of the thalamus in human reinforcement learning

Although the thalamus is supposed to be involved in reinforcement-based decision-making, there is no direct evidence regarding the involvement of this subcortical structure in humans. To fill this gap, we leveraged rare intra-thalamic electrophysiological recordings in patients and found that temporally structured thalamic oscillations encode key learning signals. Our findings also provide neural insight into the computational mechanisms of action inhibition in punishment avoidance learning.

neuroscience↗

bioRxiv2022

The functional form of value normalization in human reinforcement learning

Reinforcement learning research in humans and other species indicates that rewards are represented in a context-dependent manner. More specifically, reward representations seem to be normalized as a function of the value of the alternative options. The dominant view postulates that value context-dependence is achieved via a divisive normalization rule, inspired by perceptual decision-making research. However, behavioral and neural evidence points to another plausible mechanism: range normalization. Critically, previous experimental designs were ill-suited to disentangle the divisive and the range normalization accounts, which generate similar behavioral predictions in many circumstances. To address this question, we designed a new learning task where we manipulated, across learning contexts, the number of options and the value ranges. Behavioral and computational analyses falsify the divisive normalization account and rather provide support for the range normalization rule. Together, these results shed new light on the computational mechanisms underlying context-dependence in learning and decision-making.

animal behavior and cognition↗

bioRxiv2020

The construction and deconstruction of sub-optimal preferences through range-adapting reinforcement learning

Converging evidence suggests that economic values are rescaled as a function of the range of the available options. Critically, although locally adaptive, range adaptation has been shown to lead to suboptimal choices. This is particularly striking in reinforcement learning (RL) situations when options are extrapolated from their original context. Range adaptation can be seen as the result of an adaptive coding process aiming at increasing the signal-to-noise ratio. However, this hypothesis leads to a counter-intuitive prediction: decreasing outcome uncertainty should increase range adaptation and, consequently, extrapolation errors. Here, we tested the paradoxical relation between range adaptation and performance in a large sample of subjects performing variants of a RL task, where we manipulated task difficulty. Results confirmed that range adaptation induces systematic extrapolation errors and is stronger when decreasing outcome uncertainty. Finally, we propose a range-adapting model and show that it is able to parsimoniously capture all the observed results.

animal behavior and cognition↗

bioRxiv2020

The effect of counterfactual information on outcome value coding in medial prefrontal and cingulate cortex: from an absolute to a relative neural code.

Adaptive coding of stimuli in visual cortex is well documented in perception, where it supports efficient encoding over a broad range of possible percepts. Recently, a similar neural mechanism has been reported also in value-based decision, where it allows optimal encoding of vast ranges of values in PFC: neuronal response to value depends on the choice context (relative coding), rather than being invariant across contexts (absolute coding). Additionally, value learning is sensitive to the amount of feedback information: providing complete feedback (both obtained and forgone outcomes) instead of partial feedback (only obtained outcome) improves learning. However, it is unclear whether relative coding occurs in all PFC regions and how it is affected by feedback information. We systematically investigated univariate and multivariate feedback encoding in various PFC regions and compared three modes of neural coding: absolute, partially-adaptive and fully-adaptive. Twenty-eight human participants (both sexes) performed a learning task while undergoing fMRI scanning. On each trial, they chose between two symbols associated with a certain outcome. Then, the decision outcome was revealed. Notably, in half of the trials participants received partial feedback, while in the other half they got complete feedback. We used univariate and multivariate analysis to explore value encoding in different feedback conditions. We found that both obtained and forgone outcomes were encoded in mPFC, but with opposite sign in ventral and dorsal subdivisions. Moreover, we showed that increasing feedback information induced a switch from absolute to relative coding. Our results suggest that complete feedback information promotes context-dependent outcome encoding. SIGNIFICANCE STATEMENTThis study offers a systematic investigation of the effect of the amount of feedback information (partial vs. complete) on univariate and multivariate outcome value encoding, within multiple regions in mPFC/ACC critical for value-based decisions and behavioural adaptation. Moreover, we provide the first comparison of three possible models of neural coding (i.e., absolute, partially-adaptive, and fully-adaptive coding) of value signal in mPFC by using commensurable measures of model accuracy. Taken together, our results help build a more comprehensive picture of how the human brain encodes and processes outcome value. In particular, our results suggest that simultaneous presentation of obtained and foregone outcomes promotes relative value representation.

neuroscience↗

bioRxiv2019

Imitation as a model-free process in human reinforcement learning

While there is not doubt that social signals affect human reinforcement learning, there is still no consensus about their exact computational implementation. To address this issue, we compared three hypotheses about the algorithmic implementation of imitation in human reinforcement learning. A first hypothesis, decision biasing, postulates that imitation consists in transiently biasing the learners action selection without affecting her value function. According to the second hypothesis, model-based imitation, the learner infers the demonstrators value function through inverse reinforcement learning and uses it for action selection. Finally, according to the third hypothesis, value shaping, demonstrators actions directly affect the learners value function. We tested these three psychologically plausible hypotheses in two separate experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task, where we manipulated the quantity and the quality of the demonstrators choices. We show through model comparison that value shaping is favored, which provides a new perspective on how imitation is integrated into human reinforcement learning.

neuroscience↗

bioRxiv2019

Fluctuations in the homeostatic neural circuitry influence consistency in choices between cultural goods

Forrest Gump or Matrix? Preference-based decisions are subjective and entail self-reflection. However, these self-related features are unaccounted for by known neural mechanisms of valuation and choice. Self-related processes have been linked to a basic interoceptive biological mechanism, the neural monitoring of heartbeats, in particular in ventromedial prefrontal cortex (vmPFC), a region also involved in value encoding. We thus hypothesized a functional coupling between the neural monitoring of heartbeats and the precision of value encoding in vmPFC. Human participants were presented with pairs of movie titles. They indicated either which movie they preferred, or performed a control objective visual discrimination that did not require self-reflection. Using magnetoencephalography, we measured heartbeat-evoked responses (HERs) before option presentation, and confirmed that HERs in vmPFC were larger when preparing to the subjective, self-related task. We retrieved the expected cortical value network during choice with time-resolved statistical modeling. Crucially, we show that larger HERs before option presentation are followed by stronger value encoding during choice in vmPFC. This effect is independent of the overall vmPFC baseline activity. The neural interaction between HERs and value encoding predicted preference-based choice consistency over time, accounting for both inter-individual differences and trial-to-trial fluctuations within individuals. Neither cardiac activity nor arousal fluctuations could account for any of the effects. HERs did not interact with the encoding of perceptual evidence in the discrimination task. Our results show that the self-reflection underlying preference-based decisions involves the integration of HERs to subjective value encoding in vmPFC, and that this integration contributes to preference stability. Significance statementDeciding whether you prefer Forrest Gump or Matrix is based on subjective values, which only you, the decision-maker, can estimate and compare, by asking yourself. Yet, how self-reflection is biologically implemented and its contribution to subjective valuation are not known. We show that in ventromedial prefrontal cortex, the neural response to heartbeats, an interoceptive self-related process, influences the cortical representation of subjective value. The neural interaction between the cortical monitoring of heartbeats and value encoding predicts choice consistency, i.e. whether you consistently prefer Forrest Gump over Matrix over time. Our results pave the way for the quantification of self-related process in decision making and may shed new light on the relationship between maladaptive decisions and impaired interoception.

neuroscience↗

bioRxiv2019

Temporal Chunking as a Mechanism for Unsupervised Learning of Task-Sets

Depending on environmental demands, humans can learn and exploit multiple concurrent sets of stimulus-response associations. Mechanisms underlying the learning of such task-sets remain unknown. Here we investigate the hypothesis that task-set learning relies on unsupervised chunking of stimulus-response associations that occur in temporal proximity. We examine behavioral and neural data from a task-set learning experiment using a network model. We first show that task-set learning can be achieved provided the timescale of chunking is slower than the timescale of stimulus-response learning. Fitting the model to behavioral data confirmed this expectation and led to specific predictions linking chunking and task-set retrieval that were borne out by behavioral performance and reaction times. Comparing the model activity with BOLD signal allowed us to identify neural correlates of task-set retrieval in a functional network involving ventral and dorsal prefrontal cortex, with the dorsal system preferentially engaged when retrievals are used to improve performance.

neuroscience↗

bioRxiv2019

Choosing and learning: outcome valence differentially affects learning from free versus forced choices

Positivity bias refers to learning more from positive than negative events. This learning asymmetry could either reflect a preference for positive events in general, or be the upshot of a more general, and perhaps, ubiquitous, \"choice-confirmation\" bias, whereby agents preferentially integrate information that confirms their previous decision. We systematically compared these two theories with 3 experiments mixing free- and forced-choice conditions, featuring factual and counterfactual learning and varying action requirements across \"go\" and \"no-go\" trials. Computational analyses of learning rates showed clear and robust evidence in favour of the \"choice-confirmation\" theory: participants amplified positive prediction errors in free-choice conditions while being valence-neutral on forced-choice conditions. We suggest that a choice-confirmation bias is adaptive to the extent that it reinforces actions that are most likely to meet an individuals needs, i.e. freely chosen actions. In contrast, outcomes from unchosen actions are more likely to be treated impartially, i.e. to be assigned no special value in self-determined decisions.

neuroscience↗

bioRxiv2019

Decreased confidence in loss-avoidance contexts is a primary meta-cognitive bias of human reinforcement learning

In simple instrumental-learning tasks, humans learn to seek gains and to avoid losses equally well. Yet, two effects of valence are observed. First, decisions in loss-contexts are slower, which is consistent with the Pavlovian-instrumental transfer (PIT) hypothesis. Second, loss contexts decrease individuals confidence in their choices - a bias akin to a Pavlovian-to-metacognitive transfer (PMT). Whether these two effects are two manifestations of a single mechanism or whether they can be partially dissociated is unknown. Here, across six experiments, we attempted to disrupt the PIT effects by manipulating the mapping between decisions and actions and imposing constraints on response times (RTs). Our goal was to assess the presence of the metacognitive bias in the absence of the RT bias. Were observed both PIT and PMT despite our disruption attempts, establishing that the effects of valence on motor and metacognitive responses are very robust and replicable. Nonetheless, within- and between-individual inferences reveal that the confidence bias resists the disruption of the RT bias. Therefore, although concomitant in most cases, PMT and PIT seem to be - partly - dissociable. These results highlight new important mechanistic constraints that should be incorporated in learning models to jointly explain choice, reaction times and confidence.

animal behavior and cognition↗

bioRxiv2018

Computational noise in reward-guided learning drives behavioral variability in volatile environments

When learning the value of actions in volatile environments, humans often make seemingly irrational decisions which fail to maximize expected value. We reasoned that these non-greedy decisions, instead of reflecting information seeking during choice, may be caused by computational noise in the learning of action values. Here, using reinforcement learning (RL) models of behavior and multimodal neurophysiological data, we show that the majority of non-greedy decisions stems from this learning noise. The trial-to-trial variability of sequential learning steps and their impact on behavior could be predicted both by BOLD responses to obtained rewards in the dorsal anterior cingulate cortex (dACC) and by phasic pupillary dilation - suggestive of neuromodulatory fluctuations driven by the locus coeruleus-norepinephrine (LC-NE) system. Together, these findings indicate that most of behavioral variability, rather than reflecting human exploration, is due to the limited computational precision of reward-guided learning.

neuroscience↗

bioRxiv2018

How the level of reward awareness changes the computational and electrophysiological signatures of reinforcement learning

The extent to which subjective awareness influences reward processing, and thereby affects future decisions is currently largely unknown. In the present report, we investigated this question in a reinforcement-learning framework, combining perceptual masking, computational modeling and electroencephalographic recordings (human male and female participants). Our results indicate that degrading the visibility of the reward decreased -without completely obliterating- the ability of participants to learn from outcomes, but concurrently increased their tendency to repeat previous choices. We dissociated electrophysiological signatures evoked by the reward-based learning processes from those elicited by the reward-independent repetition of previous choices and showed that these neural activities were significantly modulated by reward visibility. Overall, this report sheds new light on the neural computations underlying reward-based learning and decision-making and highlights that awareness is beneficial for the trial-by-trial adjustment of decision-making strategies.\n\nSignificance statementThe notion of reward is strongly associated with subjective evaluation, related to conscious processes such as \"pleasure\", \"liking\" and \"wanting\". Here we show that degrading reward visibility in a reinforcement learning task decreases -without completely obliterating- the ability of participants to learn from outcomes, but concurrently increases subjects tendency to repeat previous choices. Electrophysiological recordings, in combination with computational modelling, show that neural activities were significantly modulated by reward visibility. Overall, we dissociate different neural computations underlying reward-based learning and decision-making, which highlights a beneficial role of reward awareness in adjusting decision-making strategies.

neuroscience↗

bioRxiv2018

Trading off the cost of conflict against expected rewards

Value-based decision-making involves trading off the cost associated with an action against its expected reward. Research has shown that both physical and mental effort constitute such subjective costs, biasing choices away from effortful actions, and discounting the value of obtained rewards. Facing conflicts between competing action alternatives is considered aversive, as recruiting cognitive control to overcome conflict is effortful. Yet, it remains unclear whether conflict is also perceived as a cost in value-based decisions. The present study investigated this question by embedding irrelevant distractors (flanker arrows) within a reversal-learning task, with intermixed free and instructed trials. Results showed that participants learned to adapt their choices to maximize rewards, but were nevertheless biased to follow the suggestions of irrelevant distractors. Thus, the perceived cost of being in conflict with an external suggestion could sometimes trump internal value representations. By adapting computational models of reinforcement learning, we assessed the influence of conflict at both the decision and learning stages. Modelling the decision showed that conflict was avoided when evidence for either action alternative was weak, demonstrating that the cost of conflict was traded off against expected rewards. During the learning phase, we found that learning rates were reduced in instructed, relative to free, choices. Learning rates were further reduced by conflict between an instruction and subjective action values, whereas learning was not robustly influenced by conflict between ones actions and external distractors. Our results show that the subjective cost of conflict factors into value-based decision-making, and highlights that different types of conflict may have different effects on learning about action outcomes.

neuroscience↗

bioRxiv2018

Social information impairs reward learning in depressive subjects: behavioral and computational characterization

Depression is characterized by a marked decrease in social interactions and blunted sensitivity to rewards. Surprisingly, despite the importance of social deficits in depression, non-social aspects have been disproportionally investigated. As a consequence, the cognitive mechanisms underlying atypical decision-making in social contexts in depression are poorly understood. In the present study, we investigate whether deficits in reward processing interact with the social context and how this interaction is affected by self-reported depression and anxiety symptoms. Two cohorts of subjects (discovery and replication sample: N = 50 each) took part in a task involving reward learning in a social context with different levels of social information (absent, partial and complete). Behavioral analyses revealed a specific detrimental effect of depressive symptoms - but not anxiety - on behavioral performance in the presence of social information, i.e. when participants were informed about the choices of another player. Model-based analyses further characterized the computational nature of this deficit as a negative audience effect, rather than a deficit in the way others choices and rewards are integrated in decision making. To conclude, our results shed light on the cognitive and computational mechanisms underlying the interaction between social cognition, reward learning and decision-making in depressive disorders.

neuroscience↗

bioRxiv2018

Contextual influence on confidence judgments in human reinforcement learning.

The ability to correctly estimate the probability of ones choices being correct is fundamental to optimally re-evaluate previous choices or to arbitrate between different decision strategies. Experimental evidence nonetheless suggests that this metacognitive process -referred to as a confidence judgment-is susceptible to numerous biases. We investigate the effect of outcome valence (gains or losses) on confidence while participants learned stimulus-outcome associations by trial-and-error. In two experiments, we demonstrate that participants are more confident in their choices when learning to seek gains compared to avoiding losses. Importantly, these differences in confidence were observed despite objectively equal choice difficulty and similar observed performance between those two contexts. Using computational modelling, we show that this bias is driven by the context-value, a dynamically updated estimate of the average expected-value of choice options that has previously been demonstrated to be necessary to explain equal performance in the gain and loss domain. The biasing effect of context-value on confidence, also recently observed in the context of incentivized perceptual decision-making, is therefore domain-general, with likely important functional consequences.

animal behavior and cognition↗

bioRxiv2018

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences

In economics and in perceptual decision-making contextual effects are well documented, where decision weights are adjusted as a function of the distribution of stimuli. Yet, in reinforcement learning literature whether and how contextual information pertaining to decision states is integrated in learning algorithms has received comparably little attention. Here, in an attempt to fill this gap, we investigated reinforcement learning behavior and its computational substrates in a task where we orthogonally manipulated both outcome valence and magnitude, resulting in systematic variations in state-values. Over two experiments, model comparison indicated that subjects behavior is best accounted for by an algorithm which includes both reference point-dependence and range-adaptation - two crucial features of state-dependent valuation. In addition, we found state-dependent outcome valuation to progressively emerge over time, to be favored by increasing outcome information and to be correlated with explicit understanding of the task structure. Finally, our data clearly show that, while being locally adaptive (for instance in negative valence and small magnitude contexts), state-dependent valuation comes at the cost of seemingly irrational choices, when options are extrapolated out from their original contexts.

animal behavior and cognition↗