Deficits in reinforcement learning but no link to apathy in patients with schizophrenia

Abstract Negative symptoms in schizophrenia have been linked to selective reinforcement learning deficits in the context of gains combined with intact loss-avoidance learning. Fundamental mechanisms of reinforcement learning and choice are prediction error signaling and the precise representation of reward value for future decisions. It is unclear which of these mechanisms contribute to the impairments in learning from positive outcomes observed in schizophrenia. A recent study suggested that patients with severe apathy symptoms show deficits in the representation of expected value. Considering the fundamental relevance for the understanding of these symptoms, we aimed to assess the stability of these findings across studies. Sixty-four patients with schizophrenia and 19 healthy control participants performed a probabilistic reward learning task. They had to associate stimuli with gain or loss-avoidance. In a transfer phase participants indicated valuation of the previously learned stimuli by choosing among them. Patients demonstrated an overall impairment in learning compared to healthy controls. No effects of apathy symptoms on task indices were observed. However, patients with schizophrenia learned better in the context of loss-avoidance than in the context of gain. Earlier findings were thus partially replicated. Further studies are needed to clarify the mechanistic link between negative symptoms and reinforcement learning.

Download Full-text

Influenca: a gamified assessment of value-based decision-making for longitudinal studies

10.1101/2021.04.27.441601 ◽

2021 ◽

Author(s):

Monja P. Neuser ◽

Franziska Kräutlein ◽

Anne Kühnel ◽

Vanessa Teckentrup ◽

Jennifer Svaldi ◽

...

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Learning Task ◽

Reward Learning ◽

Initial Validation ◽

Cross Platform ◽

Test Retest Reliability ◽

Ecological Momentary ◽

Momentary Assessment ◽

Effective Medication

AbstractReinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. Here, we developed an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) for repeated assessment over weeks. In this task, players have to identify the most effective medication by selecting the best option after integrating offered points with changing probabilities (according to random Gaussian walks). Participants can complete up to 31 levels with 150 trials each. To encourage replay on their preferred device, in-game screens provide feedback on the progress. Using an initial validation sample of 127 players (2904 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show low to medium intra-class correlations (ICC: 0.22-0.52), indicating substantial within- and between-subject variance. Notably, state items showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

Download Full-text

Attenuation of dopamine-modulated prefrontal value signals underlies probabilistic reward learning deficits in old age

eLife ◽

10.7554/elife.26424 ◽

2017 ◽

Vol 6 ◽

Cited By ~ 19

Author(s):

Lieke de Boer ◽

Jan Axelsson ◽

Katrine Riklund ◽

Lars Nyberg ◽

Peter Dayan ◽

...

Keyword(s):

Older Adults ◽

Prefrontal Cortex ◽

Computational Modelling ◽

Learning Task ◽

D1 Receptor ◽

Reward Learning ◽

Neural Processing ◽

Learning Deficits ◽

A Value ◽

Age Related

Probabilistic reward learning is characterised by individual differences that become acute in aging. This may be due to age-related dopamine (DA) decline affecting neural processing in striatum, prefrontal cortex, or both. We examined this by administering a probabilistic reward learning task to younger and older adults, and combining computational modelling of behaviour, fMRI and PET measurements of DA D1 availability. We found that anticipatory value signals in ventromedial prefrontal cortex (vmPFC) were attenuated in older adults. The strength of this signal predicted performance beyond age and was modulated by D1 availability in nucleus accumbens. These results uncover that a value-anticipation mechanism in vmPFC declines in aging, and that this mechanism is associated with DA D1 receptor availability.

Download Full-text

Comparison of conventional and rapid-acting antidepressants in a rodent probabilistic reversal learning task

Brain and Neuroscience Advances ◽

10.1177/2398212820907177 ◽

2020 ◽

Vol 4 ◽

pp. 239821282090717 ◽

Cited By ~ 1

Author(s):

Matthew P. Wilkinson ◽

John P. Grogan ◽

Jack R. Mellor ◽

Emma S. J. Robinson

Keyword(s):

Reinforcement Learning ◽

Reversal Learning ◽

Antidepressant Treatment ◽

Learning Task ◽

Reward Processing ◽

Reward Learning ◽

Feedback Sensitivity ◽

Rule Changes ◽

Reinforcement Learning Model ◽

Probabilistic Reversal Learning

Deficits in reward processing are a central feature of major depressive disorder with patients exhibiting decreased reward learning and altered feedback sensitivity in probabilistic reversal learning tasks. Methods to quantify probabilistic learning in both rodents and humans have been developed, providing translational paradigms for depression research. We have utilised a probabilistic reversal learning task to investigate potential differences between conventional and rapid-acting antidepressants on reward learning and feedback sensitivity. We trained 12 rats in a touchscreen probabilistic reversal learning task before investigating the effect of acute administration of citalopram, venlafaxine, reboxetine, ketamine or scopolamine. Data were also analysed using a Q-learning reinforcement learning model to understand the effects of antidepressant treatment on underlying reward processing parameters. Citalopram administration decreased trials taken to learn the first rule and increased win-stay probability. Reboxetine decreased win-stay behaviour while also decreasing the number of rule changes animals performed in a session. Venlafaxine had no effect. Ketamine and scopolamine both decreased win-stay probability, number of rule changes performed and motivation in the task. Insights from the reinforcement learning model suggested that reboxetine led animals to choose a less optimal strategy, while ketamine decreased the model-free learning rate. These results suggest that reward learning and feedback sensitivity are not differentially modulated by conventional and rapid-acting antidepressant treatment in the probabilistic reversal learning task.

Download Full-text

Sharing Experience in Multitask Reinforcement Learning

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/505 ◽

2019 ◽

Author(s):

Tung-Long Vuong ◽

Do-Van Nguyen ◽

Tai-Long Nguyen ◽

Cong-Minh Bui ◽

Hai-Dang Kieu ◽

...

Keyword(s):

Reinforcement Learning ◽

Learning Process ◽

Learning Task ◽

Learning Tasks ◽

Shared Space ◽

Multiple Tasks ◽

The Stability

In multitask reinforcement learning, tasks often have sub-tasks that share the same solution, even though the overall tasks are different. If the shared-portions could be effectively identified, then the learning process could be improved since all the samples between tasks in the shared space could be used. In this paper, we propose a Sharing Experience Framework (SEF) for simultaneously training of multiple tasks. In SEF, a confidence sharing agent uses task-specific rewards from the environment to identify similar parts that should be shared across tasks and defines those parts as shared-regions between tasks. The shared-regions are expected to guide task-policies sharing their experience during the learning process. The experiments highlight that our framework improves the performance and the stability of learning task-policies, and is possible to help task-policies avoid local optimums.

Download Full-text

Acute stress impairs reward learning in men

10.1101/2020.07.13.200568 ◽

2020 ◽

Author(s):

Joana Carvalheiro ◽

Vasco A. Conceição ◽

Ana Mesquita ◽

Ana Seara-Cardoso

Keyword(s):

Reinforcement Learning ◽

Acute Stress ◽

Learning Task ◽

Reward Learning ◽

Prediction Errors ◽

How People Learn ◽

And Control ◽

The Impact ◽

Reinforcement Learning Model ◽

Better Than

AbstractAcute stress is ubiquitous in everyday life, but the extent to which acute stress affects how people learn from the outcomes of their choices is still poorly understood. Here, we investigate how acute stress impacts reward and punishment learning in men using a reinforcement-learning task. Sixty-two male participants performed the task whilst under stress and control conditions. We observed that acute stress impaired participants’ choice performance towards monetary gains, but not losses. To unravel the mechanism(s) underlying such impairment, we fitted a reinforcement-learning model to participants’ trial-by-trial choices. Computational modeling indicated that under acute stress participants learned more slowly from positive prediction errors — when the outcomes were better than expected — consistent with stress-induced dopamine disruptions. Such mechanistic understanding of how acute stress impairs reward learning is particularly important given the pervasiveness of stress in our daily life and the impact that stress can have on our wellbeing and mental health.

Download Full-text

S79. REINFORCEMENT LEARNING ABNORMALITIES IN INDIVIDUALS AT CLINICAL HIGH RISK AND IN THE FIRST EPISODE OF PSYCHOSIS

Schizophrenia Bulletin ◽

10.1093/schbul/sbaa031.145 ◽

2020 ◽

Vol 46 (Supplement_1) ◽

pp. S64-S64

Author(s):

Raktima Datta ◽

Gregory Strauss ◽

Nina Kraguljac ◽

Sydney Howie ◽

Adrienne Lahti

Keyword(s):

Reinforcement Learning ◽

High Risk ◽

Negative Symptoms ◽

Chronic Schizophrenia ◽

First Episode ◽

Healthy Controls ◽

Clinical High Risk ◽

Cross Sectional ◽

Learning Deficits ◽

Illness Onset

Abstract Background Prior studies indicate that chronic schizophrenia (SZ) is associated with a specific profile of reinforcement learning abnormalities. These impairments are characterized by: 1) reductions in learning rate, and 2) impaired Go learning and intact NoGo learning. Furthermore, each of these deficits are associated with greater severity of negative symptoms, consistent with theoretical perspectives positing that avolition and anhedonia are associated with deficits in generating, updating, and maintaining mental representations of reward value hat are needed to guide decision-making. However, it is unclear whether these deficits extend to earlier phases of psychotic illness and when individuals are unmedicated. Methods Two studies were conducted to examine reinforcement learning deficits in earlier phases of psychosis. In study 1, participants included 35 participants with first episode psychosis (FEP) and 25 healthy controls (HC). Study 2 included 17 antipsychotic naïve individuals who met criteria for attenuated psychosis syndrome (APS) (i.e., those with a prodromal syndrome) and 18 matched healthy controls (HC). In both studies, participants completed the Temporal Utility Integration Task, a measure of probabilistic reinforcement learning that contained Go and NoGo learning blocks. Participants in the clinical groups also completed neuropsychological testing and standard clinical interviews designed to determine symptom severity and diagnosis. Results FEP displayed impaired Go learning and intact NoGo learning. In contrast, APS did not display impairments in Go or NoGo learning at the group level. Negative symptoms were not significantly associated with reinforcement learning in APS participants. However, greater impairments in Go learning were associated with increased cross-sectional risk for conversion on the NAPLS risk calculator score in the APS group. Discussion Findings provide new evidence for areas of spared and impaired reinforcement learning in early phases of psychosis. Similar to chronic SZ, FEP was associated with impaired Go learning, and intact NoGo learning. Reinforcement learning is more spared in those at clinical high-risk, except those at greatest risk for conversion, where Go learning deficits are more pronounced. These findings suggest that reinforcement learning deficits may emerge early among those who are at clinical high risk for developing psychosis and that they are already pronounced by illness onset in the first episode. Importantly, these reinforcement learning deficits do not appear to be a byproduct of illness chronicity or antipsychotic medication use, but rather a consequence of the illness itself.

Download Full-text

Group value learned through interactions with members: A reinforcement learning account

10.31234/osf.io/25bc6 ◽

2021 ◽

Author(s):

Leor M Hackel ◽

Drew Kogon ◽

David Amodio ◽

Wendy Wood

Keyword(s):

Reinforcement Learning ◽

Instrumental Learning ◽

Direct Interaction ◽

Learning Task ◽

Intergroup Bias ◽

Reward Learning ◽

Individual Group ◽

Individual Level ◽

Explicit Attitudes ◽

Group Members

How do group-based interaction tendencies form through encounters with individual group members? In three experiments, in which participants interacted with group members in a reinforcement learning task presented as a money sharing game, participants formed instrumental reward associations with individual group members through direct interaction and feedback. Results revealed that individual-level reward learning generalized to a group-based representation, as indicated in self-reported group attitudes, trait impressions, and the tendency to choose subsequent interactions with novel members of the group. Moreover, group-based reward values continued to predict interactions with novel members after controlling for explicit attitudes and impressions, suggesting that instrumental learning contributes to an implicit form of group-based choice. Experiment 3 further demonstrated that group-based reward effects on interaction choices persisted even when group reward value was no longer predicted of positive outcomes, consistent with a habit-like expression of group bias. These results demonstrate a novel process of prejudice formation based on instrumental reward learning from direct interactions with individual group members. We discuss implications for existing theories of prejudice, the role of habit in intergroup bias, and intervention strategies to reduce prejudice.

Download Full-text

Effects of Amygdala Lesions on Object-Based Versus Action-Based Learning in Macaques

Cerebral Cortex ◽

10.1093/cercor/bhaa241 ◽

2020 ◽

Vol 31 (1) ◽

pp. 529-546 ◽

Cited By ~ 1

Author(s):

Craig A Taswell ◽

Vincent D Costa ◽

Benjamin M Basile ◽

Maia S Pujara ◽

Breonda Jones ◽

...

Keyword(s):

Reinforcement Learning ◽

Reversal Learning ◽

Macaca Mulatta ◽

Learning Task ◽

Neural Systems ◽

Visual Object ◽

Learning Deficits ◽

Visual Objects ◽

Object Based ◽

Amygdala Lesions

Abstract The neural systems that underlie reinforcement learning (RL) allow animals to adapt to changes in their environment. In the present study, we examined the hypothesis that the amygdala would have a preferential role in learning the values of visual objects. We compared a group of monkeys (Macaca mulatta) with amygdala lesions to a group of unoperated controls on a two-armed bandit reversal learning task. The task had two conditions. In the What condition, the animals had to learn to select a visual object, independent of its location. And in the Where condition, the animals had to learn to saccade to a location, independent of the object at the location. In both conditions choice-outcome mappings reversed in the middle of the block. We found that monkeys with amygdala lesions had learning deficits in both conditions. Monkeys with amygdala lesions did not have deficits in learning to reverse choice-outcome mappings. Rather, amygdala lesions caused the monkeys to become overly sensitive to negative feedback which impaired their ability to consistently select the more highly valued action or object. These results imply that the amygdala is generally necessary for RL.

Download Full-text

Neuroevolution approach for reinforcement learning task applied to control system of a cart pole system

10.26678/abcm.cobem2019.cob2019-1044 ◽

2019 ◽

Author(s):

Marco Boaretto ◽

Gabriel Chaves Becchi ◽

Luiza Scapinello Aquino ◽

Aderson Cleber Pifer ◽

Helon Vicente Hultmann Ayala ◽

...

Keyword(s):

Control System ◽

Reinforcement Learning ◽

Learning Task ◽

Pole System

Download Full-text

Model-Based and Model-Free Social Cognition

10.31234/osf.io/ue6j2 ◽

2019 ◽

Author(s):

Leor M Hackel ◽

Jeffrey Jordan Berg ◽

Björn Lindström ◽

David Amodio

Keyword(s):

Reinforcement Learning ◽

Social Cognition ◽

Learning Strategies ◽

Memory Systems ◽

Learning Task ◽

Financial Advisors ◽

Model Based ◽

Model Free ◽

Systems Model ◽

Task Assessment

Do habits play a role in our social impressions? To investigate the contribution of habits to the formation of social attitudes, we examined the roles of model-free and model-based reinforcement learning in social interactions—computations linked in past work to habit and planning, respectively. Participants in this study learned about novel individuals in a sequential reinforcement learning paradigm, choosing financial advisors who led them to high- or low-paying stocks. Results indicated that participants relied on both model-based and model-free learning, such that each independently predicted choice during the learning task and self-reported liking in a post-task assessment. Specifically, participants liked advisors who could provide large future rewards as well as advisors who had provided them with large rewards in the past. Moreover, participants varied in their use of model-based and model-free learning strategies, and this individual difference influenced the way in which learning related to self-reported attitudes: among participants who relied more on model-free learning, model-free social learning related more to post-task attitudes. We discuss implications for attitudes, trait impressions, and social behavior, as well as the role of habits in a memory systems model of social cognition.

Download Full-text