Time elapsed between choices in a probabilistic task correlates with repeating the same decision

AbstractReinforcement learning causes an action that yields a positive outcome more likely to be taken in the future. Here, we investigate how the time elapsed from an action affects subsequent decisions. Groups of C57BL6/J mice were housed in IntelliCages with access to water and chow ad libitum; they also had access to bottles with a reward: saccharin solution, alcohol or a mixture of the two. The probability of receiving a reward in two of the cage corners changed between 0.9 and 0.3 every 48 h over a period of ~33 days. As expected, in most animals, the odds of repeating a corner choice were increased if that choice was previously rewarded. Interestingly, the time elapsed from the previous choice also influenced the probability of repeating the choice, and this effect was independent of previous outcome. Behavioral data were fitted to a series of reinforcement learning models. Best fits were achieved when the reward prediction update was coupled with separate learning rates from positive and negative outcomes and additionally a “fictitious” update of the expected value of the nonselected choice. Additional inclusion of a time-dependent decay of the expected values improved the fit marginally in some cases.

Download Full-text

Signed and unsigned reward prediction errors dynamically enhance learning and memory

eLife ◽

10.7554/elife.61077 ◽

2021 ◽

Vol 10 ◽

Author(s):

Nina Rouhani ◽

Yael Niv

Keyword(s):

Reinforcement Learning ◽

Locus Coeruleus ◽

Learning And Memory ◽

Learning Rate ◽

Prediction Errors ◽

Learning Models ◽

The Past ◽

Reward Prediction ◽

Midbrain Dopamine ◽

Reinforcement Learning Models

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory, and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

Download Full-text

Ventral striatum’s role in learning from gains and losses

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1809833115 ◽

2018 ◽

Vol 115 (52) ◽

pp. E12398-E12406 ◽

Cited By ~ 12

Author(s):

Craig A. Taswell ◽

Vincent D. Costa ◽

Elisabeth A. Murray ◽

Bruno B. Averbeck

Keyword(s):

Reinforcement Learning ◽

Dopamine Release ◽

Ventral Striatum ◽

Learning Task ◽

Specific Role ◽

Aversive Stimuli ◽

Learning Rates ◽

Reinforcement Learning Models ◽

Gains And Losses

Adaptive behavior requires animals to learn from experience. Ideally, learning should both promote choices that lead to rewards and reduce choices that lead to losses. Because the ventral striatum (VS) contains neurons that respond to aversive stimuli and aversive stimuli can drive dopamine release in the VS, it is possible that the VS contributes to learning about aversive outcomes, including losses. However, other work suggests that the VS may play a specific role in learning to choose among rewards, with other systems mediating learning from aversive outcomes. To examine the role of the VS in learning from gains and losses, we compared the performance of macaque monkeys with VS lesions and unoperated controls on a reinforcement learning task. In the task, the monkeys gained or lost tokens, which were periodically cashed out for juice, as outcomes for choices. They learned over trials to choose cues associated with gains, and not choose cues associated with losses. We found that monkeys with VS lesions had a deficit in learning to choose between cues that differed in reward magnitude. By contrast, monkeys with VS lesions performed as well as controls when choices involved a potential loss. We also fit reinforcement learning models to the behavior and compared learning rates between groups. Relative to controls, the monkeys with VS lesions had reduced learning rates for gain cues. Therefore, in this task, the VS plays a specific role in learning to choose between rewarding options.

Download Full-text

Ageing is associated with disrupted reinforcement learning whilst learning to help others is preserved

Nature Communications ◽

10.1038/s41467-021-24576-w ◽

2021 ◽

Vol 12 (1) ◽

Author(s):

Jo Cutler ◽

Marco K. Wittmann ◽

Ayat Abdurahman ◽

Luca D. Hargitai ◽

Daniel Drew ◽

...

Keyword(s):

Older Adults ◽

Young Adults ◽

Reinforcement Learning ◽

Model Comparison ◽

Adaptive Behaviour ◽

Detailed Model ◽

Total N ◽

Learning Rates ◽

Concern For Others ◽

Reinforcement Learning Models

AbstractReinforcement learning is a fundamental mechanism displayed by many species. However, adaptive behaviour depends not only on learning about actions and outcomes that affect ourselves, but also those that affect others. Using computational reinforcement learning models, we tested whether young (age 18–36) and older (age 60–80, total n = 152) adults learn to gain rewards for themselves, another person (prosocial), or neither individual (control). Detailed model comparison showed that a model with separate learning rates for each recipient best explained behaviour. Young adults learned faster when their actions benefitted themselves, compared to others. Compared to young adults, older adults showed reduced self-relevant learning rates but preserved prosocial learning. Moreover, levels of subclinical self-reported psychopathic traits (including lack of concern for others) were lower in older adults and the core affective-interpersonal component of this measure negatively correlated with prosocial learning. These findings suggest learning to benefit others is preserved across the lifespan with implications for reinforcement learning and theories of healthy ageing.

Download Full-text

Supplemental Material for Reconciling Reinforcement Learning Models With Behavioral Extinction and Renewal: Implications for Addiction, Relapse, and Problem Gambling

Psychological Review ◽

10.1037/0033-295x.114.3.784.supp ◽

2007 ◽

Cited By ~ 1

Keyword(s):

Reinforcement Learning ◽

Problem Gambling ◽

Learning Models ◽

Behavioral Extinction ◽

Reinforcement Learning Models

Download Full-text

Bayes factors for reinforcement-learning models of the Iowa gambling task.

Decision ◽

10.1037/dec0000040 ◽

2016 ◽

Vol 3 (2) ◽

pp. 115-131 ◽

Cited By ~ 14

Author(s):

Helen Steingroever ◽

Ruud Wetzels ◽

Eric-Jan Wagenmakers

Keyword(s):

Reinforcement Learning ◽

Iowa Gambling Task ◽

Bayes Factors ◽

Gambling Task ◽

Learning Models ◽

Reinforcement Learning Models

Download Full-text

Effects of Working Memory Capacity on the Speed and Accuracy of Learning in Reinforcement Learning Models

PsycEXTRA Dataset ◽

10.1037/e528942014-552 ◽

2014 ◽

Author(s):

Adnane Ez-Zizi ◽

Simon Farrell ◽

David Leslie

Keyword(s):

Working Memory ◽

Reinforcement Learning ◽

Working Memory Capacity ◽

Memory Capacity ◽

Learning Models ◽

Reinforcement Learning Models ◽

Speed And Accuracy

Download Full-text

Supplemental Material for Reinforcement Learning Models of Risky Choice and the Promotion of Risk-Taking by Losses Disguised as Wins in Rats

Journal of Experimental Psychology Animal Learning and Cognition ◽

10.1037/xan0000141.supp ◽

2017 ◽

Keyword(s):

Reinforcement Learning ◽

Risk Taking ◽

Risky Choice ◽

Learning Models ◽

Losses Disguised As Wins ◽

Reinforcement Learning Models

Download Full-text

Individual differences in experienced and observational decision-making illuminate interactions between reinforcement learning and declarative memory

Scientific Reports ◽

10.1038/s41598-021-85322-2 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Batel Yifrah ◽

Ayelet Ramaty ◽

Genela Morris ◽

Avi Mendelsohn

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Declarative Memory ◽

Contextual Information ◽

Memory Performance ◽

Relevant Information ◽

Subjective Memory ◽

Types Of Information ◽

Reinforcement Learning Models ◽

Implicit And Explicit

AbstractDecision making can be shaped both by trial-and-error experiences and by memory of unique contextual information. Moreover, these types of information can be acquired either by means of active experience or by observing others behave in similar situations. The interactions between reinforcement learning parameters that inform decision updating and memory formation of declarative information in experienced and observational learning settings are, however, unknown. In the current study, participants took part in a probabilistic decision-making task involving situations that either yielded similar outcomes to those of an observed player or opposed them. By fitting alternative reinforcement learning models to each subject, we discerned participants who learned similarly from experience and observation from those who assigned different weights to learning signals from these two sources. Participants who assigned different weights to their own experience versus those of others displayed enhanced memory performance as well as subjective memory strength for episodes involving significant reward prospects. Conversely, memory performance of participants who did not prioritize their own experience over others did not seem to be influenced by reinforcement learning parameters. These findings demonstrate that interactions between implicit and explicit learning systems depend on the means by which individuals weigh relevant information conveyed via experience and observation.

Download Full-text