Pramipexole Enhances Reward Learning by Preserving Value Estimates

Background: Dopamine D2-like receptor agonists show promise as treatments for depression. They are thought to act by altering how individuals learn from rewarding experiences. However, the nature of these reward learning alterations, and the mechanisms by which they are produced is not clear. Reinforcement learning accounts describe three distinct processes that may produce similar changes in reward learning behaviour; increased reward sensitivity, increased inverse decision temperature and decreased value decay. As these processes produce equivalent effects on behaviour, arbitrating between them requires measurement of how expectations and prediction errors are altered. In the present study, we characterised the behavioural effects of a sustained 2-week course of the D2/3/4 receptor agonist pramipexole on reward learning and used fMRI measures of expectation and prediction error to assess which of these three mechanistic processes were responsible for the behavioural effects. Methods: 40 healthy volunteers (Age: 18-43, 50% female) were randomly allocated to receive either two weeks of pramipexole (titrated to 1mg/day) or placebo in a double-blind, between subject design. Participants completed a probabilistic instrumental learning task, in which stimuli were associated with either rewards or losses, before the pharmacological intervention and twice between days 12-15 of the intervention (once with and once without fMRI). Both asymptotic choice accuracy, and a reinforcement learning model, were used to assess reward learning. Results: Behaviourally, pramipexole specifically increased choice accuracy in the reward condition, with no effect in the loss condition. Pramipexole increased the BOLD response in the orbital frontal cortex during the expectation of win trials but decreased the BOLD response to reward prediction errors in the ventromedial prefrontal cortex. This pattern of results indicates that pramipexole enhances choice accuracy by reducing the decay of estimated values during reward learning. Conclusions: The D2-like receptor agonist pramipexole enhances reward learning by preserving learned values. This is a plausible candidate mechanism for pramipexoles observed anti-depressant effect.

Reward learning and statistical learning independently influence attentional priority of salient distractors in visual search

Existing research demonstrates different ways in which attentional prioritization of salient nontarget stimuli is shaped by prior experience: Reward learning renders signals of high-value outcomes more likely to capture attention than signals of low-value outcomes, whereas statistical learning can produce attentional suppression of the location in which salient distractor items are likely to appear. The current study combined manipulations of the value and location associated with salient distractors in visual search to investigate whether these different effects of selection history operate independently or interact to determine overall attentional prioritization of salient distractors. In Experiment 1, high-value and low-value distractors most frequently appeared in the same location; in Experiment 2, high-value and low-value distractors typically appeared in distinct locations. In both experiments, effects of distractor value and location were additive, suggesting that attention-promoting effects of value and attention-suppressing effects of statistical location-learning independently modulate overall attentional priority. Our findings are consistent with a view that sees attention as mediated by a common priority map that receives and integrates separate signals relating to physical salience and value, with signal suppression based on statistical learning determined by physical salience, but not incentive salience.

Investigation of reward learning and feedback sensitivity in non-clinical participants with a history of early life stress

Early life stress (ELS) is an important risk factor for the development of depression. Impairments in reward learning and feedback sensitivity are suggested to be an intermediate phenotype in depression aetiology therefore we hypothesised that healthy adults with a history of ELS would exhibit reward processing deficits independent of any current depressive symptoms. We recruited 64 adults with high levels of ELS and no diagnosis of a current mental health disorder and 65 controls. Participants completed the probabilistic reversal learning task and probabilistic reward task followed by depression, anhedonia, social status, and stress scales. Participants with high levels of ELS showed decreased positive feedback sensitivity in the probabilistic reversal learning task compared to controls. High ELS participants also trended towards possessing a decreased model-free learning rate. This was coupled with a decreased learning ability in the acquisition phase of block 1 following the practice session. Neither group showed a reward induced response bias in the probabilistic reward task however high ELS participants exhibited decreased stimuli discrimination. Overall, these data suggest that healthy participants without a current mental health diagnosis but with high levels of ELS show deficits in positive feedback sensitivity and reward learning in the probabilistic reversal learning task that are distinct from depressed patients. These deficits may be relevant to increased depression vulnerability.

Switching costs in stochastic environments drive the emergence of matching behaviour in animal decision-making through the promotion of reward learning strategies

A principle of choice in animal decision-making named probability matching (PM) has long been detected in animals, and can arise from different decision-making strategies. Little is known about how environmental stochasticity may influence the switching time of these different decision-making strategies. Here we address this problem using a combination of behavioral and theoretical approaches, and show, that although a simple Win-Stay-Loss-Shift (WSLS) strategy can generate PM in binary-choice tasks theoretically, budgerigars (Melopsittacus undulates) actually apply a range of sub-tactics more often when they are expected to make more accurate decisions. Surprisingly, budgerigars did not get more rewards than would be predicted when adopting a WSLS strategy, and their decisions also exhibited PM. Instead, budgerigars followed a learning strategy based on reward history, which potentially benefits individuals indirectly from paying lower switching costs. Furthermore, our data suggest that more stochastic environments may promote reward learning through significantly less switching. We suggest that switching costs driven by the stochasticity of an environmental niche can potentially represent an important selection pressure associated with decision-making that may play a key role in driving the evolution of complex cognition in animals.

The effects of atorvastatin on emotional processing, reward learning, verbal memory and inflammation in healthy volunteers: An experimental medicine study

Background: Growing evidence from clinical trials and epidemiological studies suggests that statins can have clinically significant antidepressant effects, potentially related to anti-inflammatory action on several neurobiological structures. However, the underlying neuropsychological mechanisms of these effects remain unexplored. Aims: In this experimental medicine trial, we investigated the 7-day effects of the lipophilic statin, atorvastatin on a battery of neuropsychological tests and inflammation in healthy volunteers. Methods: Fifty healthy volunteers were randomised to either 7 days of atorvastatin 20 mg or placebo in a double-blind design. Participants were assessed with psychological questionnaires and a battery of well-validated behavioural tasks assessing emotional processing, which is sensitive to putative antidepressant effects, reward learning and verbal memory, as well as the inflammatory marker, C-reactive protein. Results: Compared to placebo, 7-day atorvastatin increased the recognition ( p = 0.006), discriminability ( p = 0.03) and misclassifications ( p = 0.04) of fearful facial expression, independently from subjective states of mood and anxiety, and C-reactive protein levels. Otherwise, atorvastatin did not significantly affect any other psychological and behavioural measure, nor peripheral C-reactive protein. Conclusions: Our results reveal for the first time the early influence of atorvastatin on emotional cognition by increasing the processing of anxiety-related stimuli (i.e. increased recognition, discriminability and misclassifications of fearful facial expression) in healthy volunteers, in the absence of more general effects on negative affective bias. Further studies exploring the effects of statins in depressed patients, especially with raised inflammatory markers, may clarify this finding and inform future clinical trials.

Smoking as a Common Modulator of Sensory Gating and Reward Learning in Individuals with Psychotic Disorders

Motivational and perceptual disturbances co-occur in psychosis and have been linked to aberrations in reward learning and sensory gating, respectively. Although traditionally studied independently, when viewed through a predictive coding framework, these processes can both be linked to dysfunction in striatal dopaminergic prediction error signaling. This study examined whether reward learning and sensory gating are correlated in individuals with psychotic disorders, and whether nicotine—a psychostimulant that amplifies phasic striatal dopamine firing—is a common modulator of these two processes. We recruited 183 patients with psychotic disorders (79 schizophrenia, 104 psychotic bipolar disorder) and 129 controls and assessed reward learning (behavioral probabilistic reward task), sensory gating (P50 event-related potential), and smoking history. Reward learning and sensory gating were correlated across the sample. Smoking influenced reward learning and sensory gating in both patient groups; however, the effects were in opposite directions. Specifically, smoking was associated with improved performance in individuals with schizophrenia but impaired performance in individuals with psychotic bipolar disorder. These findings suggest that reward learning and sensory gating are linked and modulated by smoking. However, disorder-specific associations with smoking suggest that nicotine may expose pathophysiological differences in the architecture and function of prediction error circuitry in these overlapping yet distinct psychotic disorders.

Distinct replay signatures for planning and memory maintenance

Theories of neural replay propose that it supports a range of different functions, most prominently planning and memory maintenance. Here, we test the hypothesis that distinct replay signatures relate to planning and memory maintenance. Our reward learning task required human participants to utilize structure knowledge for 'model-based' evaluation, while maintaining knowledge for two independent and randomly alternating task environments. Using magnetoencephalography (MEG) and multivariate analysis, we found neural evidence for compressed forward replay during planning and backward replay following reward feedback. Prospective replay strength was enhanced for the current environment when the benefits of a model-based planning strategy were higher. Following reward receipt, backward replay for the alternative, distal environment was enhanced as a function of decreasing recency of experience for that environment. Consistent with a memory maintenance role, stronger maintenance-related replay was associated with a modulation of subsequent choices. These findings identify distinct replay signatures consistent with key theoretical proposals on planning and memory maintenance functions, with their relative strength modulated by on-going computational and task demands.

Where is the money? Dynamics in feedback processing and attention during spatial probabilistic learning

Learned feature-based stimulus-reward-associations can modulate behavior and the underlying neural processing of information. In our study, we investigated the neurocognitive mechanisms underlying learning of spatial stimulus-reward-associations. Participants performed a probabilistic spatial reward-learning task that required participants, within 40 trials, to learn which out of four locations on a computer screen yielded the most gain-feedback when chosen. Our behavioral findings show that participants learned to choose which location was most rewarding. Those findings were paralleled by significant amplitude differences in event-related potentials (ERPs) elicited by the presentation of loss and gain feedback; the amplitude of the feedback-related negativity (FRN) was more negative in response to loss feedback compared to gain feedback, but showed no modulation by trial-number. On the other hand, the late positive component (LPC), became larger in response to losses as the learning-set progressed, but smaller in response to gains. Additionally, immediately following feedback presentation, brain activity in the visual cortex - read out through alpha frequency oscillations measured over occipital sites - was predictive of the amplitude of the N2pc ERP component, a marker of spatial attention orienting, observed on the next trial. Taken together, we elucidated neurocognitive dynamics underlying feedback processing in spatial reward learning, and the subsequent effects that spatial stimulus-reward association learning have on spatial attention.