Biology subjects

Collins, A. G. E.

Publications and source records attributed to Collins, A. G. E..

17 recordsLinked to original sources

bioRxiv2026

Learning to Pursue Goals that Conflict with Prior Experience Relies on Fast Revaluation and Slow Value Updating

Humans can flexibly pursue novel goals, learning to reach them from goal-dependent teaching signals. However, the breadth of this flexibility and its underlying mechanisms remain unclear. For example, can humans flexibly learn to reach a typically avoided outcome (e.g., +0) while avoiding a typically sought one (+1)? We hypothesized that humans are sufficiently flexible to learn from counterintuitive goals, but at a cost, and this ability is supported by complementary temporal dynamics of rapid cognitive control and slower, iterative outcome value updating. In two experiments, we confirmed that participants learned when pursuing counterintuitive goals, but less effectively than standard goals. Furthermore, extended practice reduced performance gaps. To characterize the underlying mechanisms, we developed a novel computational model incorporating fast goal switching and slow outcome value updating. This model best captured group-level behavior; individuals differed in the efficiency of these processes, capturing differences in the cost initially incurred by counterintuitive goals. Crucially, a stronger initial cost did not permanently limit performance: slower value updating compensated for early inefficiencies in rapid adaptation. Furthermore, comparing early and late goal-switch learning revealed that greater goal alignment was associated with faster outcome processing, suggesting a transition from effortful control to efficient outcome evaluation. Thus, despite individual differences in initial flexibility, repeated attempts to reach counterintuitive goals decreased initial biases. More broadly, our findings show that goal-dependent outcome valuation leverages complementary fast and slow processes to enable flexible and efficient goal pursuit over time.

neuroscience↗

bioRxiv2025

Dual process impairments in reinforcement learning and working memory systems underlie learning deficits in physiological anxiety

Anxiety has been robustly linked to deficits in frontal executive function including working memory (WM) and attentional control processes. However, although anxiety has also been associated with impaired performance on learning tasks, computational investigations of reinforcement learning (RL) impairment in anxiety have yielded mixed results. WM processes are known to contribute to learning behavior in parallel to RL processes and to modulate the effective learning rate as a function of load. However, WM processes have typically not been modeled in investigations of anxiety and RL. In the current study, we leveraged an experimental paradigm (RLWM) which manipulates the relative contributions of WM and RL processes in a reinforcement learning and retention task using multiple stimulus set sizes. Using a computational model of interactive RL and WM processes, we investigated whether individual differences in physiological or cognitive anxiety impacted task performance via deficits in RL or WM. Elevated physiological, but not cognitive, anxiety scores were strongly associated with worse performance during learning and retention testing across all set sizes. Computationally, higher physiological anxiety scores were significantly related to reduced learning rate and increased rate of WM decay. To highlight the importance of modeling WM contributions to learning, we considered the effect of fitting RL models without WM modules to the data. Here we found that reduced learning performance for higher physiological anxiety was at least partially misattributed to stochastic decision noise in 9 out of 10 RL-only models considered. These findings reveal a dual-process impairment in learning in anxiety that is linked to a more physiological than cognitive anxiety phenotype. More broadly, this work also points to the importance of accounting for the contribution of WM to RL when investigating psychopathology-related deficits in learning.

neuroscience↗

bioRxiv2025

Genetic changes linked to two different syndromic forms of autism enhance reinforcement learning in adolescent male but not female mice

Autism Spectrum Disorder (ASD) is characterized by restricted and repetitive behaviors and social differences, both of which may manifest, in part, from underlying differences in corticostriatal circuits and reinforcement learning. Here, we investigated reinforcement learning in developing mice with mutations in either Tsc2 or Shank3, both high-confidence ASD risk genes associated with major syndromic forms of ASD. Using an odor-based two-alternative forced choice (2AFC) task, we tested early adolescent mice of both sexes and found male Tsc2 and Shank3B heterozygote (Het) mice showed enhanced learning performance compared to their wild type (WT) siblings. No gain of function was observed in females. Using a novel reinforcement learning (RL) based computational model to infer learning rate as well as policy-level task engagement and disengagement, we found that the gain of function in males was driven by an enhanced positive learning rate in both Tsc2 and Shank3B Het mice. The gain of function in Het males was absent when mice were trained with a probabilistic reward schedule. These findings in two ASD mouse models reveal a convergent learning phenotype that shows similar sensitivity to sex and environmental uncertainty. These data can inform our understanding of both strengths and challenges associated with autism, while providing further evidence that sex and experience of uncertainty modulate autism-related phenotypes. Significance StatementReinforcement learning is a foundational form of learning that is widely used in behavioral interventions for autism. Here, we measured reinforcement learning in developing mice carrying genetic mutations linked to two different syndromic forms of autism. We found that males showed strengths in reinforcement learning compared to their wild type siblings, while females showed no differences. This gain of function in males was no longer observed when uncertainty was introduced into the reward schedule for correct choices. These findings support a model in which diverse genetic changes interact with sex to generate common phenotypes underlying autism. Our data further support the idea that autism risk genes may produce strengths as well as challenges in behavioral function.

neuroscience↗

bioRxiv2024

Adolescent and adult mice use both incremental reinforcement learning and short term memory when learning concurrent stimulus-action associations

Computational modeling has revealed that human research participants use both rapid working memory (WM) and incremental reinforcement learning (RL) (RL+WM) to solve a simple instrumental learning task, relying on WM when the number of stimuli is small and supplementing with RL when the number of stimuli exceeds WM capacity. Inspired by this work, we examined which learning systems and strategies are used by adolescent and adult mice when they first acquire a conditional associative learning task. In a version of the human RL+WM task translated for rodents, mice were required to associate odor stimuli (from a set of 2 or 4 odors) with a left or right port to receive reward. Using logistic regression and computational models to analyze the first 200 trials per odor, we determined that mice used both incremental RL and stimulus-insensitive, one-back strategies to solve the task. While these one-back strategies may be a simple form of short-term or working memory, they did not approximate the boost to learning performance that has been observed in human participants using WM in a comparable task. Adolescent and adult mice also showed comparable performance, with no change in learning rate or softmax beta parameters with adolescent development and task experience. However, reliance on a one-back perseverative, win-stay strategy increased with development in males in both odor set sizes. Our findings advance a simple conditional associative learning task and new models to enable the isolation and quantification of reinforcement learning alongside other strategies mice use while learning to associate stimuli with rewards within a single behavioral session. These data and methods can inform and aid comparative study of reinforcement learning across species. Author summaryHere we studied the strategies and mechanisms mice use to learn a simple two choice odor based task in a single session. Using a set size manipulation and computational models we find evidence that mice use incremental reinforcement learning as well as several short-term (one-back) strategies to earn water reward. Our data and models clarify how mice learn a simple task and establish methods by which mouse and human reinforcement learning may be isolated for cross-species comparison of learning.

neuroscience↗

bioRxiv2024

Striatal Dopamine Can Enhance Learning, Both Fast and Slow, and Also Make it Cheaper

Associations can be learned incrementally, via reinforcement learning (RL), or stored instantly in working memory (WM). While WM is fast, it is also capacity-limited and effortful. Striatal dopamine may promote RL plasticity, and WM, by facilitating updating and effort exertion. Yet, prior studies have failed to distinguish between dopamines effects on RL versus WM. N = 100 participants completed a paradigm isolating these systems in a double-blind study measuring dopamine synthesis with [18F]-FDOPA imaging and manipulating dopamine with methylphenidate and sulpiride. Learning is enhanced among high synthesis capacity individuals and by methylphenidate, but impaired by sulpiride. Methylphenidate also blunts effort cost learning. Computational modeling reveals that individuals with high dopamine synthesis rely more on WM, while methylphenidate boosts their RL rates. The D2 antagonist sulpiride reduces accuracy due to diminished WM involvement and faster WM decay. We conclude that dopamine enhances both slow RL, and fast WM, by promoting plasticity and reducing effort sensitivity. These results also highlight the need to control for dopamines effects on WM when testing its effects on RL.

animal behavior and cognition↗

bioRxiv2023

Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning

Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison among a wide set of cognitive models based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation contributes to instrumental task performance in mice and is reflected in mesolimbic dopamine signaling.

animal behavior and cognition↗

bioRxiv2023

Artificial neural networks for model identification and parameter estimation in computational cognitive models

1Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate. 2 Author SummaryComputational cognitive models occupy an important position in cognitive science research, as they offer a simple way of quantifying cognitive processes (such as how fast someone learns, or how noisy they are in choice selection), and testing which cognitive theories offer a better explanation of the behavior. To relate cognitive models to the behavioral data, researchers rely on statistical tools that require estimating the likelihood of observed data under the assumptions of the cognitive model. This is, however, not possible to do for all models as some models present significant challenges to likelihood computation. In this work, we use artificial neural networks (ANNs) to bypass likelihood computation and approximation altogether, and demonstrate the success of this approach applied to model parameter estimation and model comparison. The proposed method is a contribution to ongoing development of modeling tools which will enable cognitive researchers to test a broader range of theories of cognition.

animal behavior and cognition↗

bioRxiv2022

Age-related decline in prefrontal glutamate predicts failure to efficiently deploy working memory in the service of learning

The ability to use past experience to effectively guide decision making declines in older adulthood. Such declines have been theorized to emerge from either impairments of striatal reinforcement learning systems (RL) or impairments of recurrent networks in prefrontal and parietal cortex that support working memory (WM). Distinguishing between these hypotheses has been challenging because either RL or WM could be used to facilitate successful decision making in typical laboratory tasks. Here we investigated the neurocomputational correlates of age-related decision making deficits using an RL-WM task to disentangle these mechanisms, a computational model to quantify them, and magnetic resonance spectroscopy to link them to their molecular bases. Our results reveal that task performance is worse in older age, in a manner best explained by working memory deficits, as might be expected if cortical recurrent networks were unable to sustain persistent activity across multiple trials. Consistent with this, we show that older adults had lower levels of prefrontal glutamate, the excitatory neurotransmitter thought to support persistent activity, compared to younger adults. Individuals with the lowest prefrontal glutamate levels displayed the greatest impairments in working memory after controlling for other anatomical and metabolic factors. Together, our results suggest that lower levels of prefrontal glutamate may contribute to failures of working memory systems and impaired decision making in older adulthood.

animal behavior and cognition↗

bioRxiv2021

Plucking a string or playing a G? Degree of choice abstraction impacts human reinforcement learning

In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values which change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (e.g. pressing a key with the index finger), to more general choices that can be executed in a number of ways (e.g. getting dinner at the restaurant). But does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: participants were slower and less accurate in learning correct choices of a general format compared to learning more concrete, motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: the values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response is not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1, by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL learning. In both experiments we ruled out the explanation that the difference in performance between two condition types was driven by difficulty/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.

animal behavior and cognition↗

bioRxiv2021

Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context

Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g., tasks, models) and that they capture interpretable (i.e., unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8-30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. However, generalization was significantly below ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g., reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models. HighlightsO_LIComputational cognitive modeling often makes two implicit assumptions: 1) Model parameters generalize between studies and models. 2) Parameters are interpretable, i.e., neurally, cognitively, and/or mechanistically specific. C_LIO_LIWe empirically investigate these assumptions in Reinforcement Learning (RL), using a large developmental dataset featuring three different learning tasks in a within-participant design. C_LIO_LIWe find that no parameters generalize between tasks in terms of absolute parameter values. However, RL decision noise/exploration parameters generalize in terms of between-participant variation, showing similar age trajectories across tasks. Interpretability was low for all parameters, especially those related to learning and memory. C_LIO_LIEven for parameters that showed generalization, the degree of generalization was significantly lower than in an artificial sample with perfect generalization for most parameters, revealing a relative lack of generalization. C_LIO_LIThese findings are in accordance with previous claims about the developmental trajectory of decision noise/exploration parameters, but suggest that claims about the development of learning rates or Forgetting might not be generalizable: these parameters are unlikely to capture the same neuro-cognitive processes across different tasks. C_LIO_LIWe suggest ways forward to improve the generalizability and interpretability of model parameters, in particular by focusing more on task context. C_LI

neuroscience↗

bioRxiv2020

Modeling Changes in Probabilistic Reinforcement Learning during Adolescence

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggests probabilistic learning may be inefficient in youth compared to adults [1], while others suggest it may be more efficient in youth that are in mid adolescence [2, 3]. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time horizon); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development. Author summaryAdolescence is a time of great uncertainty. It is also a critical time for brain development, learning, and decision making in social and educational domains. There are currently contradictory findings about learning in adolescence. We sought to better isolate how learning from stable probabilistic contingencies changes during adolescence with a task that previously showed interesting results in adolescents. We collected a relatively large sample size (297 participants) across a wide age range (8-30), to trace the adolescent developmental trajectory of learning under stable but uncertain conditions. We found that age in our sample was positively associated with higher learning rates and lower choice exploration. Within narrow age bins, we found that higher saliva testosterone levels were associated with higher learning rates in participants age 13-15 years. These findings can help us better isolate the trajectory of maturation of core learning and decision making processes during adolescence.

neuroscience↗

bioRxiv2020

Executive modulation of brain reward systems endows goals with value

People often learn from the outcomes of their actions, even when these outcomes do not involve material rewards or punishments. How does our brain provide this flexibility? We combined behavior, computational modeling, and functional neuroimaging to probe whether learning from transient goals harnesses the same circuitry that supports learning from secondary reinforcers. Behavior and neuroimaging revealed that "one-shot" transient goals (abstract fractal images seen once) can act as a substitute for rewards during instrumental learning, and produce reliable reward-like signals in dopaminergic reward circuits. Moreover, we found evidence that prefrontal correlates of executive control may play a role in shaping these responses in reward circuits. These results suggest that learning from abstract goal outcomes is supported by an interplay between high-level representations in prefrontal cortex and low-level responses in subcortical reward circuits. This interaction may allow humans to perform reinforcement learning over flexible, arbitrarily abstract reward functions.

neuroscience↗

bioRxiv2020

Understanding the Unique Advantage of Adolescents in Stochastic, Volatile Environments: Combining Reinforcement Learning and Bayesian Inference

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated whether adolescents are uniquely adapted to this transition, compared to younger children and adults. In a stochastic, volatile reversal-learning task with a sample of 291 participants aged 8-30, we found that adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants mental models peaked closest to optimal in adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonously with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that may be specifically adapted for volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.

neuroscience↗

bioRxiv2020

Temporal and state abstractions for efficient learning, transfer and composition in humans

Humans use prior knowledge to efficiently solve novel tasks, but how they structure past knowledge to enable such fast generalization is not well understood. We recently proposed that hierarchical state abstraction enabled generalization of simple one-step rules, by inferring context clusters for each rule. However, humans daily tasks are often temporally extended, and necessitate more complex multi-step, hierarchically structured strategies. The options framework in hierarchical reinforcement learning provides a theoretical framework for representing such transferable strategies. Options are abstract multi-step policies, assembled from simpler one-step actions or other options, that can represent meaningful reusable strategies as temporal abstractions. We developed a novel sequential decision making protocol to test if humans learn and transfer multi-step options. In a series of four experiments, we found transfer effects at multiple hierarchical levels of abstraction that could not be explained by flat reinforcement learning models or hierarchical models lacking temporal abstraction. We extended the options framework to develop a quantitative model that blends temporal and state abstractions. Our model captures the transfer effects observed in human participants. Our results provide evidence that humans create and compose hierarchical options, and use them to explore in novel contexts, consequently transferring past knowledge and speeding up learning.

neuroscience↗

bioRxiv2019

Computational evidence for hierarchically-structured reinforcement learning in humans

Humans have the fascinating ability to achieve goals in a complex and constantly changing world, still surpassing modern machine learning algorithms in terms of flexibility and learning speed. It is generally accepted that a crucial factor for this ability is the use of abstract, hierarchical representations, which employ structure in the environment to guide learning and decision making. Nevertheless, how we create and use these hierarchical representations is poorly understood. This study presents evidence that human behavior can be characterized as hierarchical reinforcement learning (RL). We designed an experiment to test specific predictions of hierarchical RL using a series of subtasks in the realm of context-based learning, and observed several behavioral markers of hierarchical RL, such as asymmetric switch costs between changes in higher-level versus lower-level features, faster learning in higher-valued compared to lower-valued contexts, and preference for higher-valued compared to lower-valued contexts. We replicated these results across three independent samples. We simulated three models: a classic RL, a hierarchical RL, and a hierar-chical Bayesian model, and compared their behavior to human results. While the flat RL model captured some aspects of participants sensitivity to outcome values, and the hierarchical Bayesian model some markers of transfer, only hierarchical RL accounted for all patterns observed in human behavior. This work shows that hierarchical RL, a biologically-inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments, and opens the avenue for future research in this field.

neuroscience↗

bioRxiv2019

Choice suppression through opponent but not independent function of the striatal indirect pathway

The dorsomedial striatum (DMS) plays a key role in action selection, but little is known about how direct and indirect pathway spiny projection neurons (dSPNs and iSPNs) contribute to choice suppression in freely moving animals. Here, we used pathway-specific chemogenetic manipulation during a serial choice foraging task to test opposing predictions for iSPN function generated by two theories: 1) the select/suppress heuristic which suggests iSPN activity is required to suppress alternate choices and 2) the network-inspired Opponent Actor Learning model (OpAL) which proposes that the weighted difference of dSPN and iSPN activity determines choice. We found that chemogenetic activation, but not inhibition, of iSPNs disrupted learned suppression of nonrewarded choices, consistent with the predictions of the OpAL model. Our findings suggest that iSPNs role in stopping and freezing does not extend in a simple fashion to choice suppression. These data may provide insights critical for the successful design of interventions for addiction or other conditions in which suppression of behavior is desirable.

neuroscience↗

bioRxiv2017

Cross-task contributions of fronto-basal ganglia circuitry in response inhibition and conflict-induced slowing

Why are we so slow in choosing the lesser of two evils? We considered whether such slowing relates to uncertainty about the value of these options, which arises from the tendency to avoid them during learning, and whether such slowing relates to fronto-subthalamic inhibitory control mechanisms. 49 participants performed a reinforcement-learning task and a stop-signal task while fMRI was recorded. A reinforcement-learning model was used to quantify learning strategies. Individual differences in lose-lose slowing related to information uncertainty due to sampling, and independently, to less efficient response inhibition in the stop-signal task. Neuroimaging analysis revealed an analogous dissociation: subthalamic nucleus (STN) BOLD activity related to variability in stopping latencies, whereas weaker fronto-subthalamic connectivity related to slowing and information sampling. Across tasks, fast inhibitors increased STN activity for successfully cancelled responses in the stop task, but decreased activity for lose-lose choices. These data support the notion that fronto-STN communication implements a rapid but transient brake on response execution, and that slowing due to decision uncertainty could result from an inefficient release of this \"hold your horses\" mechanism.

neuroscience↗