Biology subjects

Sutlief, E.

Publications and source records attributed to Sutlief, E..

3 recordsLinked to original sources

bioRxiv2025

Reward-reset interval timing drives patch foraging decisions through neural state transitions in dorsomedial striatum

Activities with temporally distributed returns require deciding not only what to do, but when to stop. Although optimal foraging theory predicts that patch departure should depend on the total time invested in a patch, behavior across species often reflects sensitivity to recent rewards. The dorsomedial striatum (DMS) mediates both goal-directed decision-making and interval timing, two functions that converge during patch foraging, where animals must decide when to exit patches to maximize reward rates. We recorded extracellular activity from neurons in DMS while freely moving mice performed a patch-foraging task. Mice employed a reward-reset strategy, primarily basing their exit decisions on the time since the last reward, with systematic adjustments for patch residence time and environmental reward-rate context. Individual neurons in DMS underwent discrete firing rate transitions at characteristic delays following each reward. These transition times tiled the post-reward interval across the population, producing an accumulation-to-threshold signal that reset with each reward and whose rate was sensitive to the cost of time, increasing with higher environmental reward rates and as rewards occurred later in the patch. This accumulation reached a consistent threshold at the moment of patch exit, encoding the animals intended stopping time. Together, these results identify a time-cost-variable, reward-reset, accumulation-to-threshold computation in DMS that integrates environmental reward rate, elapsed time from pursuit engagement, and elapsed time from reward occurrence to determine when to abandon a depleting pursuit.

neuroscience↗

bioRxiv2024

The value of initiating a pursuit in temporal decision-making

Reward-rate maximization is a prominent normative principle commonly held in behavioral ecology, neuroscience, economics, and artificial intelligence. Here, we identify and compare equations for evaluating the worth of initiating pursuits that an agent could implement to enable reward-rate maximization. We identify two fundamental temporal decision-making categories requiring the valuation of the initiation of a pursuit--forgo and choice decision-making--over which we generalize and analyze the optimal solution for how to evaluate a pursuit in order to maximize reward rate. From this reward-rate-maximizing formulation, we derive expressions for the subjective value of a pursuit, i.e. that pursuits equivalent immediate reward magnitude, and reveal that times cost is composed of an apportionment, in addition to, an opportunity cost. By re-expressing subjective value as a temporal discounting function, we show precisely how the temporal discounting function of a reward-rate-optimal agent is sensitive not just to the properties of a considered pursuit, but to the time spent and reward acquired outside of the pursuit for every instance spent within it. In doing so, we demonstrate how the apparent discounting function of a reward-rate-optimizing agent depends on the temporal structure of the environment and is a combination of hyperbolic and linear components, whose contributions relate the apportionment and opportunity cost of time, respectively. We further then show how purported signs of suboptimal behavior (hyperbolic discounting, the Delay effect, the Magnitude effect, the Sign effect) are in fact consistent with reward-rate maximization. Having clarified what features are and are not signs of optimal decision-making, we analyze the impact of the misestimation of reward rate-maximizing parameters in order to better account for the pattern of errors actually observed in humans and animals. We find that error in agents assessment of the apportionment of time that underweights the time spent outside versus inside a considered pursuit type is the likely driver of suboptimal temporal decision-making observed behaviorally. We term this the Malapportionment Hypothesis. This generalized form for reward-rate maximization and its relation to subjective value and temporal discounting allows the true pattern of errors exhibited by humans and animals to be more deeply understood, identified, and quantified, which is key to deducing the learning algorithms and representational architectures actually used by humans and animals to evaluate the worth of pursuits.

neuroscience↗

bioRxiv2019

A history-derived reward prediction error signal in ventral pallidum

Learning from past interactions with the environment is critical for adaptive behavior. Within the framework of reinforcement learning, the nervous system builds expectations about future reward by computing reward prediction errors (RPEs), the difference between actual and predicted rewards. Correlates of RPEs have been observed in the midbrain dopamine system, which is thought to locally compute this important variable in service of learning. However, the extent to which RPE signals may be computed upstream of the dopamine system is largely unknown. Here, we quantify history-based RPE signals in the ventral pallidum (VP), an input region to the midbrain dopamine system implicated in reward-seeking behavior. We trained rats to associate cues with future delivery of reward and fit computational models to predict individual neuron firing rates at the time of reward delivery. We found that a subset of VP neurons encoded RPEs and did so more robustly than nucleus accumbens, an input to VP. VP RPEs predicted trial-by-trial task engagement, and optogenetic inhibition of VP reduced subsequent task-related reward seeking. Consistent with reinforcement learning, activity of VP RPE cells adapted when rewards were delivered in blocks. We further found that history- and cue-based RPEs were largely separate across the VP neural population. The presence of behaviorally-instructive RPE signals in the VP suggests a pivotal role for this region in value-based computations.

neuroscience↗