scholarly journals Basal forebrain-derived acetylcholine encodes valence-free reinforcement prediction error

2020 ◽  
Author(s):  
JF Sturgill ◽  
P Hegedus ◽  
SJ Li ◽  
Q Chevy ◽  
A Siebels ◽  
...  
Keyword(s):  
Basal Forebrain ◽  
Prediction Error ◽  
Learning Task ◽  
Cellular Targets ◽  
Functional Roles ◽  
Cholinergic Basal Forebrain ◽  
Reward Prediction ◽  
Signaling Mechanisms ◽  
Dopaminergic Systems ◽  
Serial Reversal

AbstractAcetylcholine (Ach) is released by the cholinergic basal forebrain (CBF) throughout the cortical mantle and is implicated in behavioral functions ranging from arousal to attention to learning. Yet what signal ACh provides to cortex remains unresolved, hindering our understanding of its functional roles. Here we demonstrate that the CBF signals unsigned reinforcement prediction error, in contrast to dopamine (DA) neurons that encode reward prediction error. We show that both CBF neuronal activity and acetylcholine (ACh) release at cortical targets signal reinforcement delivery, acquire responses to predictive stimuli and show diminished responses to expected outcomes, hallmarks of a prediction error. To compare ACh with DA, we simultaneously monitored the activity of both neuromodulators during a serial reversal learning task. ACh tracked learning as swiftly as DA during acquisition but lagged slightly during extinction, suggesting that these neuromodulators play complementary roles in reinforcement as their patterns of innervation, cellular targets, and signaling mechanisms are themselves complementary. Through retrograde viral tracing we show that the cholinergic and dopaminergic systems engage overlapping upstream circuits, accounting for their coordination during learning. This predictive and valence-free signal explains how ACh can proactively and retroactively improve the processing of behaviorally important stimuli, be they good or bad.

scholarly journals Frontal Theta Oscillatory Activity Is a Common Mechanism for the Computation of Unexpected Outcomes and Learning Rate

2014 ◽  
Vol 26 (3) ◽  
pp. 447-458 ◽  
Author(s):  
Ernest Mas-Herrero ◽  
Josep Marco-Pallarés
Keyword(s):  
Prediction Error ◽  
Learning Task ◽  
Learning Rate ◽  
Oscillatory Activity ◽  
Common Mechanism ◽  
Prediction Errors ◽  
Reward Prediction Error ◽  
Reward Prediction ◽  
Medial Pfc ◽  
The Impact

In decision-making processes, the relevance of the information yielded by outcomes varies across time and situations. It increases when previous predictions are not accurate and in contexts with high environmental uncertainty. Previous fMRI studies have shown an important role of medial pFC in coding both reward prediction errors and the impact of this information to guide future decisions. However, it is unclear whether these two processes are dissociated in time or occur simultaneously, suggesting that a common mechanism is engaged. In the present work, we studied the modulation of two electrophysiological responses associated to outcome processing—the feedback-related negativity ERP and frontocentral theta oscillatory activity—with the reward prediction error and the learning rate. Twenty-six participants performed two learning tasks differing in the degree of predictability of the outcomes: a reversal learning task and a probabilistic learning task with multiple blocks of novel cue–outcome associations. We implemented a reinforcement learning model to obtain the single-trial reward prediction error and the learning rate for each participant and task. Our results indicated that midfrontal theta activity and feedback-related negativity increased linearly with the unsigned prediction error. In addition, variations of frontal theta oscillatory activity predicted the learning rate across tasks and participants. These results support the existence of a common brain mechanism for the computation of unsigned prediction error and learning rate.

scholarly journals Differential reinforcement encoding along the hippocampal long axis helps resolve the explore–exploit dilemma

2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Alexandre Y. Dombrovski ◽  
Beatriz Luna ◽  
Michael N. Hallquist
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Differential Reinforcement ◽  
Cognitive Maps ◽  
Learning Task ◽  
Natural Environments ◽  
Reward Prediction Error ◽  
Reward Function ◽  
Reward Prediction ◽  
Reward Information

Abstract When making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Here we report that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation on a reinforcement learning task with a spatially structured reward function. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.

scholarly journals Momentary subjective well-being depends on learning and not reward

eLife ◽  
2020 ◽  
Vol 9 ◽  
Author(s):  
Bastien Blain ◽  
Robb B Rutledge
Keyword(s):  
Prediction Error ◽  
Computational Modelling ◽  
Well Being ◽  
Learning Task ◽  
Adaptive Behaviour ◽  
Subjective Well Being ◽  
Reward Prediction Error ◽  
Reward Prediction ◽  
The Difference ◽  
Momentary Happiness

Subjective well-being or happiness is often associated with wealth. Recent studies suggest that momentary happiness is associated with reward prediction error, the difference between experienced and predicted reward, a key component of adaptive behaviour. We tested subjects in a reinforcement learning task in which reward size and probability were uncorrelated, allowing us to dissociate between the contributions of reward and learning to happiness. Using computational modelling, we found convergent evidence across stable and volatile learning tasks that happiness, like behaviour, is sensitive to learning-relevant variables (i.e. probability prediction error). Unlike behaviour, happiness is not sensitive to learning-irrelevant variables (i.e. reward prediction error). Increasing volatility reduces how many past trials influence behaviour but not happiness. Finally, depressive symptoms reduce happiness more in volatile than stable environments. Our results suggest that how we learn about our world may be more important for how we feel than the rewards we actually receive.

scholarly journals Role of Reversal Learning Impairment in Social Disinhibition following Severe Traumatic Brain Injury

2016 ◽  
Vol 22 (3) ◽  
pp. 303-313 ◽  
Author(s):  
Katherine Osborne-Crowley ◽  
Skye McDonald ◽  
Jacqueline A. Rushby
Keyword(s):  
Traumatic Brain Injury ◽  
Brain Injury ◽  
Reversal Learning ◽  
Prediction Error ◽  
Learning Task ◽  
Reward Prediction Error ◽  
Reward Prediction ◽  
The Social ◽  
Learning Impairment ◽  
Reversal Errors

AbstractObjectives: The current study aimed to determine whether reversal learning impairments and feedback-related negativity (FRN), reflecting reward prediction error signals generated by negative feedback during the reversal learning tasks, were associated with social disinhibition in a group of participants with traumatic brain injury (TBI). Methods: Number of reversal errors on a social and a non-social reversal learning task and FRN were examined for 21 participants with TBI and 21 control participants matched for age. Participants with TBI were also divided into low and high disinhibition groups based on rated videotaped interviews. Results: Participants with TBI made more reversal errors and produced smaller amplitude FRNs than controls. Furthermore, participants with TBI high on social disinhibition made more reversal errors on the social reversal learning task than did those low on social disinhibition. FRN amplitude was not related to disinhibition. Conclusions: These results suggest that impairment in the ability to update behavior when social reinforcement contingencies change plays a role in social disinhibition after TBI. Furthermore, the social reversal learning task used in this study may be a useful neuropsychological tool for detecting susceptibility to acquired social disinhibition following TBI. Finally, that the FRN amplitude was not associated with social disinhibition suggests that reward prediction error signals are not critical for behavioral adaptation in the social domain. (JINS, 2016, 21, 303–313)

scholarly journals Behavioural and computational evidence for memory consolidation biased by reward-prediction errors

2019 ◽  
Author(s):  
Emma L. Roscow ◽  
Matthew W. Jones ◽  
Nathan F. Lepora
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Learning Task ◽  
Male Rats ◽  
Adaptive Behaviour ◽  
Prediction Errors ◽  
Reward Prediction Error ◽  
Reward Prediction ◽  
Per Se ◽  
Reinforcement Learning Model

AbstractNeural activity encoding recent experiences is replayed during sleep and rest to promote consolidation of the corresponding memories. However, precisely which features of experience influence replay prioritisation to optimise adaptive behaviour remains unclear. Here, we trained adult male rats on a novel maze-based rein-forcement learning task designed to dissociate reward outcomes from reward-prediction errors. Four variations of a reinforcement learning model were fitted to the rats’ behaviour over multiple days. Behaviour was best predicted by a model incorporating replay biased by reward-prediction error, compared to the same model with no replay; random replay or reward-biased replay produced poorer predictions of behaviour. This insight disentangles the influences of salience on replay, suggesting that reinforcement learning is tuned by post-learning replay biased by reward-prediction error, not by reward per se. This work therefore provides a behavioural and theoretical toolkit with which to measure and interpret replay in striatal, hippocampal and neocortical circuits.

scholarly journals Differential reinforcement encoding along the hippocampal long axis helps resolve the explore/exploit dilemma

2020 ◽  
Author(s):  
Alexandre Y. Dombrovski ◽  
Beatriz Luna ◽  
Michael N. Hallquist
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Differential Reinforcement ◽  
Cognitive Maps ◽  
Learning Task ◽  
Natural Environments ◽  
Reward Prediction Error ◽  
Reward Function ◽  
Reward Prediction ◽  
Reward Information

ABSTRACTWhen making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Using a reinforcement learning task with a spatially structured reward function, we show that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.

Sign in / Sign up

Export Citation Format

Share Document