Ageing is associated with disrupted reinforcement learning whilst learning to help others is preserved

AbstractReinforcement learning is a fundamental mechanism displayed by many species. However, adaptive behaviour depends not only on learning about actions and outcomes that affect ourselves, but also those that affect others. Using computational reinforcement learning models, we tested whether young (age 18–36) and older (age 60–80, total n = 152) adults learn to gain rewards for themselves, another person (prosocial), or neither individual (control). Detailed model comparison showed that a model with separate learning rates for each recipient best explained behaviour. Young adults learned faster when their actions benefitted themselves, compared to others. Compared to young adults, older adults showed reduced self-relevant learning rates but preserved prosocial learning. Moreover, levels of subclinical self-reported psychopathic traits (including lack of concern for others) were lower in older adults and the core affective-interpersonal component of this measure negatively correlated with prosocial learning. These findings suggest learning to benefit others is preserved across the lifespan with implications for reinforcement learning and theories of healthy ageing.

Download Full-text

Ageing disrupts reinforcement learning whilst learning to help others is preserved

10.1101/2020.12.02.407718 ◽

2020 ◽

Author(s):

Jo Cutler ◽

Marco Wittmann ◽

Ayat Abdurahman ◽

Luca Hargitai ◽

Daniel Drew ◽

...

Keyword(s):

Older Adults ◽

Reinforcement Learning ◽

Model Comparison ◽

Computational Modelling ◽

Psychopathic Traits ◽

Learning Ability ◽

Adaptive Behaviour ◽

Detailed Model ◽

Total N ◽

Learning Rates

AbstractReinforcement learning is a fundamental mechanism displayed by many species from mice to humans. However, adaptive behaviour depends not only on learning associations between actions and outcomes that affect ourselves, but critically, also outcomes that affect other people. Existing studies suggest reinforcement learning ability declines across the lifespan and self-relevant learning can be computationally separated from learning about rewards for others, yet how older adults learn what rewards others is unknown. Here, using computational modelling of a probabilistic reinforcement learning task, we tested whether young (age 18-36) and older (age 60-80, total n=152) adults can learn to gain rewards for themselves, another person (prosocial), or neither individual (control). Detailed model comparison showed that a computational model with separate learning rates best explained how people learn associations for different recipients. Young adults were faster to learn when their actions benefitted themselves, compared to when they helped others. Strikingly however, older adults showed reduced self-bias, with a relative increase in the rate at which they learnt about actions that helped others, compared to themselves. Moreover, we find evidence that these group differences are associated with changes in psychopathic traits over the lifespan. In older adults, psychopathic traits were significantly reduced and negatively correlated with prosocial learning rates. Importantly, older people with the lowest levels of psychopathy had the highest prosocial learning rates. These findings suggest learning how our actions help others is preserved across the lifespan with implications for our understanding of reinforcement learning mechanisms and theoretical accounts of healthy ageing.

Download Full-text

Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices

Social Cognitive and Affective Neuroscience ◽

10.1093/scan/nsaa089 ◽

2020 ◽

Vol 15 (6) ◽

pp. 695-707 ◽

Cited By ~ 4

Author(s):

Lei Zhang ◽

Lukas Lengersdorff ◽

Nace Mikus ◽

Jan Gläscher ◽

Claus Lamm

Keyword(s):

Reinforcement Learning ◽

Prediction Error ◽

Model Comparison ◽

Hierarchical Modeling ◽

Learning Rate ◽

Neural Activation ◽

Affective Neuroscience ◽

Outcome Valence ◽

Practical Guidelines ◽

Reinforcement Learning Models

Abstract The recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla–Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Download Full-text

Using reinforcement learning models in social neuroscience: frameworks, pitfalls, and suggestions of best practices

10.31234/osf.io/uthw2 ◽

2019 ◽

Author(s):

Lei Zhang ◽

Lukas Lengersdorff ◽

Nace Mikus ◽

Jan Gläscher ◽

Claus Lamm

Keyword(s):

Reinforcement Learning ◽

Prediction Error ◽

Model Comparison ◽

Hierarchical Modeling ◽

Learning Rate ◽

Neural Activation ◽

Affective Neuroscience ◽

Outcome Valence ◽

Practical Guidelines ◽

Reinforcement Learning Models

Recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla-Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Download Full-text

Ventral striatum’s role in learning from gains and losses

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1809833115 ◽

2018 ◽

Vol 115 (52) ◽

pp. E12398-E12406 ◽

Cited By ~ 12

Author(s):

Craig A. Taswell ◽

Vincent D. Costa ◽

Elisabeth A. Murray ◽

Bruno B. Averbeck

Keyword(s):

Reinforcement Learning ◽

Dopamine Release ◽

Ventral Striatum ◽

Learning Task ◽

Specific Role ◽

Aversive Stimuli ◽

Learning Rates ◽

Reinforcement Learning Models ◽

Gains And Losses

Adaptive behavior requires animals to learn from experience. Ideally, learning should both promote choices that lead to rewards and reduce choices that lead to losses. Because the ventral striatum (VS) contains neurons that respond to aversive stimuli and aversive stimuli can drive dopamine release in the VS, it is possible that the VS contributes to learning about aversive outcomes, including losses. However, other work suggests that the VS may play a specific role in learning to choose among rewards, with other systems mediating learning from aversive outcomes. To examine the role of the VS in learning from gains and losses, we compared the performance of macaque monkeys with VS lesions and unoperated controls on a reinforcement learning task. In the task, the monkeys gained or lost tokens, which were periodically cashed out for juice, as outcomes for choices. They learned over trials to choose cues associated with gains, and not choose cues associated with losses. We found that monkeys with VS lesions had a deficit in learning to choose between cues that differed in reward magnitude. By contrast, monkeys with VS lesions performed as well as controls when choices involved a potential loss. We also fit reinforcement learning models to the behavior and compared learning rates between groups. Relative to controls, the monkeys with VS lesions had reduced learning rates for gain cues. Therefore, in this task, the VS plays a specific role in learning to choose between rewarding options.

Download Full-text

Time elapsed between choices in a probabilistic task correlates with repeating the same decision

10.1101/643965 ◽

2019 ◽

Author(s):

Judyta Jabłońska ◽

Łukasz Szumiec ◽

Piotr Zieliński ◽

Jan Rodriguez Parkitna

Keyword(s):

Reinforcement Learning ◽

Saccharin Solution ◽

Positive Outcome ◽

Time Dependent ◽

Reward Prediction ◽

Learning Rates ◽

Access To Water ◽

Expected Values ◽

Ad Libitum ◽

Reinforcement Learning Models

AbstractReinforcement learning causes an action that yields a positive outcome more likely to be taken in the future. Here, we investigate how the time elapsed from an action affects subsequent decisions. Groups of C57BL6/J mice were housed in IntelliCages with access to water and chow ad libitum; they also had access to bottles with a reward: saccharin solution, alcohol or a mixture of the two. The probability of receiving a reward in two of the cage corners changed between 0.9 and 0.3 every 48 h over a period of ~33 days. As expected, in most animals, the odds of repeating a corner choice were increased if that choice was previously rewarded. Interestingly, the time elapsed from the previous choice also influenced the probability of repeating the choice, and this effect was independent of previous outcome. Behavioral data were fitted to a series of reinforcement learning models. Best fits were achieved when the reward prediction update was coupled with separate learning rates from positive and negative outcomes and additionally a “fictitious” update of the expected value of the nonselected choice. Additional inclusion of a time-dependent decay of the expected values improved the fit marginally in some cases.

Download Full-text

Task-Switching Training and Transfer

Journal of Psychophysiology ◽

10.1027/0269-8803/a000189 ◽

2018 ◽

Vol 32 (3) ◽

pp. 106-130 ◽

Cited By ~ 5

Author(s):

Zsófia Anna Gaál ◽

István Czigler

Keyword(s):

Older Adults ◽

Young Adults ◽

Cognitive Training ◽

Task Switching ◽

Difficulty Level ◽

Training Procedure ◽

Far Transfer ◽

Near Transfer ◽

Related Potentials ◽

Transfer Tasks

Abstract. We used task-switching (TS) paradigms to study how cognitive training can compensate age-related cognitive decline. Thirty-nine young (age span: 18–25 years) and 40 older (age span: 60–75 years) women were assigned to training and control groups. The training group received 8 one-hour long cognitive training sessions in which the difficulty level of TS was individually adjusted. The other half of the sample did not receive any intervention. The reference task was an informatively cued TS paradigm with nogo stimuli. Performance was measured on reference, near-transfer, and far-transfer tasks by behavioral indicators and event-related potentials (ERPs) before training, 1 month after pretraining, and in case of older adults, 1 year later. The results showed that young adults had better pretraining performance. The reference task was too difficult for older adults to form appropriate representations as indicated by the behavioral data and the lack of P3b components. But after training older adults reached the level of performance of young participants, and accordingly, P3b emerged after both the cue and the target. Training gain was observed also in near-transfer tasks, and partly in far-transfer tasks; working memory and executive functions did not improve, but we found improvement in alerting and orienting networks, and in the execution of variants of TS paradigms. Behavioral and ERP changes remained preserved even after 1 year. These findings suggest that with an appropriate training procedure older adults can reach the level of performance seen in young adults and these changes persist for a long period. The training also affects the unpracticed tasks, but the transfer depends on the extent of task similarities.

Download Full-text

Inefficient Encoding as an Explanation for Age-Related Deficits in Recollection-Based Processing

Journal of Psychophysiology ◽

10.1027/0269-8803/a000122 ◽

2014 ◽

Vol 28 (3) ◽

pp. 148-161 ◽

Cited By ~ 12

Author(s):

David Friedman ◽

Ray Johnson

Keyword(s):

Older Adults ◽

Young Adults ◽

Cognitive Processes ◽

Brain Activity ◽

Memory Performance ◽

Brain Regions ◽

Semantic Retrieval ◽

Age Related ◽

The Face ◽

Episodic Memories

A cardinal feature of aging is a decline in episodic memory (EM). Nevertheless, there is evidence that some older adults may be able to “compensate” for failures in recollection-based processing by recruiting brain regions and cognitive processes not normally recruited by the young. We review the evidence suggesting that age-related declines in EM performance and recollection-related brain activity (left-parietal EM effect; LPEM) are due to altered processing at encoding. We describe results from our laboratory on differences in encoding- and retrieval-related activity between young and older adults. We then show that, relative to the young, in older adults brain activity at encoding is reduced over a brain region believed to be crucial for successful semantic elaboration in a 400–1,400-ms interval (left inferior prefrontal cortex, LIPFC; Johnson, Nessler, & Friedman, 2013 ; Nessler, Friedman, Johnson, & Bersick, 2007 ; Nessler, Johnson, Bersick, & Friedman, 2006 ). This reduced brain activity is associated with diminished subsequent recognition-memory performance and the LPEM at retrieval. We provide evidence for this premise by demonstrating that disrupting encoding-related processes during this 400–1,400-ms interval in young adults affords causal support for the hypothesis that the reduction over LIPFC during encoding produces the hallmarks of an age-related EM deficit: normal semantic retrieval at encoding, reduced subsequent episodic recognition accuracy, free recall, and the LPEM. Finally, we show that the reduced LPEM in young adults is associated with “additional” brain activity over similar brain areas as those activated when older adults show deficient retrieval. Hence, rather than supporting the compensation hypothesis, these data are more consistent with the scaffolding hypothesis, in which the recruitment of additional cognitive processes is an adaptive response across the life span in the face of momentary increases in task demand due to poorly-encoded episodic memories.

Download Full-text