Sex differences in behavioral responding and dopamine release during early Pavlovian learning

Learning associations between cues and rewards requires the mesolimbic dopamine system. The dopamine response to cues signals differences in reward value in well-trained animals. These value-related dopamine responses are absent during early learning when cues signal differences in the reward rate, which suggests cue-evoked dopamine release conveys differences between outcomes only after extensive training. However, it is unclear if this lack of value coding by cue-evoked dopamine release during early learning is unique to when cues signal differences in reward rate, or if this is also evident when cues signal differences in other value-related parameters such as reward size. To address this, we utilized a Pavlovian conditioning task in which one audio cue was associated with a small reward (one pellet) and another audio cue was associated with a large reward (three pellets). We performed fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens of male and female rats throughout early learning. Cue-evoked dopamine release did not encode differences in reward value, and there were no differences in this response between males and females. However, female rats exhibited higher levels of conditioned responding and a faster latency to respond. Reward-evoked dopamine release scaled with reward size in both sexes, though there were transient sex differences in the dynamics of this response. We additionally identified sex differences in the number of post-reward head entries. Collectively these data illustrate sustained sex differences in behavioral responding as well as transient sex differences in reward-evoked dopamine release.

Seizing the Opportunity: Lifespan Differences in the Effects of the Opportunity Cost of Time on Cognitive Control

Previous work suggests that lifespan developmental differences in cognitive control reflect maturational and aging-related changes in prefrontal cortex functioning. However, complementary explanations exist: It could be that children and older adults differ from younger adults in how they balance the effort of engaging in control against its potential benefits. Here we test whether the degree of cognitive effort expenditure depends on the opportunity cost of time (average reward rate per unit time): if the average reward rate is high, participants should withhold cognitive effort whereas if it is low, they should invest more. In Experiment 1, we examine this hypothesis in children, adolescents, younger, and older adults, by applying a reward rate manipulation in two cognitive control tasks: a modified Erikson Flanker and a task-switching paradigm. We found that young adults and adolescents reflexively withheld effort when the opportunity cost of time was high, whereas older adults and, to a lesser degree children, invested more resources to accumulate reward as quickly as possible. We tentatively interpret these results in terms of age- and task-specific differences in the processing of the opportunity cost of time. We qualify our findings in a second experiment in younger adults in which we address an alternative explanation of our results and show that the observed age differences in effort expenditure may not result from differences in task difficulty. To conclude, we think that our results present an interesting first step at relating opportunity costs to motivational processes across the lifespan. We frame the implications of further work in this area within a recent developmental model of resource-rationality, which points to developmental sweet spots in cognitive control.

Rat anterior cingulate cortex continuously signals decision variables in a patch foraging task

In patch foraging tasks, animals must decide whether to remain with a depleting resource or to leave it in search of a potentially better source of reward. In such tasks, animals consistently follow the general predictions of optimal foraging theory (the Marginal Value Theorem; MVT): to leave a patch when the reward rate in the current patch depletes to the average reward rate across patches. Prior studies implicate an important role for the anterior cingulate cortex (ACC) in foraging decisions based on MVT: within single trials, ACC activity increases immediately preceding foraging decisions, and across trials, these dynamics are modulated as the value of staying in the patch depletes to the average reward rate. Here, we test whether these activity patterns reflect dynamic encoding of decision-variables and whether these signals are directly involved in decision-making or serve a more general function such as monitoring task performance or allocating cognitive control. We developed a leaky accumulator model based on the MVT that generates estimates of decision variables within and across trials, and tested model predictions against ACC activity recorded from rats performing a patch foraging task. Model predicted changes in MVT decision variables closely matched rat ACC activity. Next, we pharmacologically inactivated ACC to test the contribution of these signals to decision-making. Despite ACC inactivation, rats still followed the MVT decision rule, suggesting that foraging decision variables represented in the ACC are used for a more general function such as regulating cognitive control or motivation.

Dopamine and reward-related vigor in younger and older human participants

Vigor reflects how motivated one is to respond to a stimulus. We previously showed that humans are more vigorous when more reward is available on average, and that this relationship is modulated by the dopamine precursor levodopa. Dopamine signalling and probabilistic reward learning degrade with age, so the relationship between vigor and reward should change with age. We test this and assess whether the relationship between vigor and reward correlates with D1 dopamine receptor availability measured using Positron Emission Tomography. We measured response times of 30 older and 30 younger subjects during an oddball discrimination task where rewards varied systematically between trial. Reward rate had a similar impact on the vigor of both groups. We observed a weak positive association across subjects between ventral striatal dopamine receptor availability and effect of average reward rate on response time, which was in the opposite direction to our prediction. Overall, the effect of reward on response vigor is similar between younger and older humans and is weakly sensitive to dopamine D1 receptor availability.

Control over patch encounters changes foraging behaviour

SummaryForaging is a common decision problem in natural environments. When new exploitable sites are always available, a simple optimal strategy is to leave a current site when its return falls below a single average reward rate. Here, we examined foraging in a more structured environment, with a limited number of sites that replenished at different rates and had to be revisited. When participants could choose sites, they visited fast-replenishing sites more often, left sites at higher levels of reward, and achieved a higher net reward rate. Decisions to exploit-or-leave a site were best explained with a computational model estimating separate reward rates for each site. This suggests option-specific information can be used to construct a threshold for patch leaving in some foraging settings, rather than a single average reward rate.

Reinforcement with Fading Memories

We study the effect of imperfect memory on decision making in the context of a stochastic sequential action-reward problem. An agent chooses a sequence of actions, which generate discrete rewards at different rates. She is allowed to make new choices at rate β, whereas past rewards disappear from her memory at rate μ. We focus on a family of decision rules where the agent makes a new choice by randomly selecting an action with a probability approximately proportional to the amount of past rewards associated with each action in her memory. We provide closed form formulas for the agent’s steady-state choice distribution in the regime where the memory span is large ([Formula: see text]) and show that the agent’s success critically depends on how quickly she updates her choices relative to the speed of memory decay. If [Formula: see text], the agent almost always chooses the best action (that is, the one with the highest reward rate). Conversely, if [Formula: see text], the agent chooses an action with a probability roughly proportional to its reward rate.

Signatures of processing complexity during global cognitive states in ventromedial prefrontal cortex

ABSTRACTBehavioral neuroscience almost exclusively studies behavior during tasks and ignores the unstructured inter-trial interval (ITI). However, it is unlikely that the ITI is simply an idling or paused mode; instead, it is a likely time for globally focused cognition, in which attention is disengaged from the task at hand and oriented more broadly. To gain insight into the computational underpinnings of globally focused cognition, we recorded from neurons in a core decision-making region, area 14 of ventromedial prefrontal cortex (vmPFC), as macaques performed a foraging search task with long inter-trial intervals (ITIs). We find that during the ITI, ensemble firing is associated with increased discriminability of a key mnemonic variable, recent reward rate, which in turn predicts upcoming search strategy. ITI activity is also associated with increased ensemble dimensionality and faster subspace reorganization, presumed markers of processing complexity. These results demonstrate the flexible nature of mnemonic processing and support the idea that the brain makes use of ostensible downtime to engage in complex processing.