scholarly journals Pupil responses as indicators of value-based decision-making

2018 ◽  
Author(s):  
Joanne C. Van Slooten ◽  
Sara Jahfari ◽  
Tomas Knapen ◽  
Jan Theeuwes
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Computational Model ◽  
Cognitive Processes ◽  
Learning Task ◽  
Brain Regions ◽  
Computational Approach ◽  
Prediction Errors ◽  
Reward Prediction ◽  
Exciting Possibility

AbstractPupil responses have been used to track cognitive processes during decision-making. Studies have shown that in these cases the pupil reflects the joint activation of many cortical and subcortical brain regions, also those traditionally implicated in value-based learning. However, how the pupil tracks value-based decisions and reinforcement learning is unknown. We combined a reinforcement learning task with a computational model to study pupil responses during value-based decisions, and decision evaluations. We found that the pupil closely tracks reinforcement learning both across trials and participants. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs. After feedback, early dilation scaled with value uncertainty, whereas later constriction scaled with reward prediction errors. Our computational approach systematically implicates the pupil in value-based decisions, and the subsequent processing of violated value beliefs, ttese dissociable influences provide an exciting possibility to non-invasively study ongoing reinforcement learning in the pupil.

scholarly journals Neural Signatures of Prediction Errors in a Decision-Making Task Are Modulated by Action Execution Failures

2018 ◽  
Author(s):  
Samuel D. McDougle ◽  
Peter A. Butcher ◽  
Darius Parvin ◽  
Fasial Mushtaq ◽  
Yael Niv ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Learning Task ◽  
Prediction Errors ◽  
Action Execution ◽  
Model Driven ◽  
Assignment Of Credit ◽  
Level Information ◽  
High Level

AbstractDecisions must be implemented through actions, and actions are prone to error. As such, when an expected outcome is not obtained, an individual should not only be sensitive to whether the choice itself was suboptimal, but also whether the action required to indicate that choice was executed successfully. The intelligent assignment of credit to action execution versus action selection has clear ecological utility for the learner. To explore this scenario, we used a modified version of a classic reinforcement learning task in which feedback indicated if negative prediction errors were, or were not, associated with execution errors. Using fMRI, we asked if prediction error computations in the human striatum, a key substrate in reinforcement learning and decision making, are modulated when a failure in action execution results in the negative outcome. Participants were more tolerant of non-rewarded outcomes when these resulted from execution errors versus when execution was successful but the reward was withheld. Consistent with this behavior, a model-driven analysis of neural activity revealed an attenuation of the signal associated with negative reward prediction error in the striatum following execution failures. These results converge with other lines of evidence suggesting that prediction errors in the mesostriatal dopamine system integrate high-level information during the evaluation of instantaneous reward outcomes.

Dorsal Striatal–midbrain Connectivity in Humans Predicts How Reinforcements Are Used to Guide Decisions

2009 ◽  
Vol 21 (7) ◽  
pp. 1332-1345 ◽  
Author(s):  
Thorsten Kahnt ◽  
Soyoung Q Park ◽  
Michael X Cohen ◽  
Anne Beck ◽  
Andreas Heinz ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Functional Connectivity ◽  
Ventral Striatum ◽  
Critical Role ◽  
Prediction Errors ◽  
Midbrain Neurons ◽  
Reward Prediction ◽  
Future Behavior ◽  
The Impact

It has been suggested that the target areas of dopaminergic midbrain neurons, the dorsal (DS) and ventral striatum (VS), are differently involved in reinforcement learning especially as actor and critic. Whereas the critic learns to predict rewards, the actor maintains action values to guide future decisions. The different midbrain connections to the DS and the VS seem to play a critical role in this functional distinction. Here, subjects performed a dynamic, reward-based decision-making task during fMRI acquisition. A computational model of reinforcement learning was used to estimate the different effects of positive and negative reinforcements on future decisions for each subject individually. We found that activity in both the DS and the VS correlated with reward prediction errors. Using functional connectivity, we show that the DS and the VS are differentially connected to different midbrain regions (possibly corresponding to the substantia nigra [SN] and the ventral tegmental area [VTA], respectively). However, only functional connectivity between the DS and the putative SN predicted the impact of different reinforcement types on future behavior. These results suggest that connections between the putative SN and the DS are critical for modulating action values in the DS according to both positive and negative reinforcements to guide future decision making.

scholarly journals Neural basis of decision making guided by emotional outcomes

2015 ◽  
Vol 113 (9) ◽  
pp. 3056-3068 ◽  
Author(s):  
Kentaro Katahira ◽  
Yoshi-Taka Matsuda ◽  
Tomomi Fujimura ◽  
Kenichi Ueno ◽  
Takeshi Asamizuya ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Brain Regions ◽  
Parahippocampal Gyrus ◽  
Prediction Errors ◽  
Neural Basis ◽  
Decision Outcomes ◽  
Gain Loss ◽  
Emotional Events

Emotional events resulting from a choice influence an individual's subsequent decision making. Although the relationship between emotion and decision making has been widely discussed, previous studies have mainly investigated decision outcomes that can easily be mapped to reward and punishment, including monetary gain/loss, gustatory stimuli, and pain. These studies regard emotion as a modulator of decision making that can be made rationally in the absence of emotions. In our daily lives, however, we often encounter various emotional events that affect decisions by themselves, and mapping the events to a reward or punishment is often not straightforward. In this study, we investigated the neural substrates of how such emotional decision outcomes affect subsequent decision making. By using functional magnetic resonance imaging (fMRI), we measured brain activities of humans during a stochastic decision-making task in which various emotional pictures were presented as decision outcomes. We found that pleasant pictures differentially activated the midbrain, fusiform gyrus, and parahippocampal gyrus, whereas unpleasant pictures differentially activated the ventral striatum, compared with neutral pictures. We assumed that the emotional decision outcomes affect the subsequent decision by updating the value of the options, a process modeled by reinforcement learning models, and that the brain regions representing the prediction error that drives the reinforcement learning are involved in guiding subsequent decisions. We found that some regions of the striatum and the insula were separately correlated with the prediction error for either pleasant pictures or unpleasant pictures, whereas the precuneus was correlated with prediction errors for both pleasant and unpleasant pictures.

scholarly journals Behavioural and computational evidence for memory consolidation biased by reward-prediction errors

2019 ◽  
Author(s):  
Emma L. Roscow ◽  
Matthew W. Jones ◽  
Nathan F. Lepora
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Learning Task ◽  
Male Rats ◽  
Adaptive Behaviour ◽  
Prediction Errors ◽  
Reward Prediction Error ◽  
Reward Prediction ◽  
Per Se ◽  
Reinforcement Learning Model

AbstractNeural activity encoding recent experiences is replayed during sleep and rest to promote consolidation of the corresponding memories. However, precisely which features of experience influence replay prioritisation to optimise adaptive behaviour remains unclear. Here, we trained adult male rats on a novel maze-based rein-forcement learning task designed to dissociate reward outcomes from reward-prediction errors. Four variations of a reinforcement learning model were fitted to the rats’ behaviour over multiple days. Behaviour was best predicted by a model incorporating replay biased by reward-prediction error, compared to the same model with no replay; random replay or reward-biased replay produced poorer predictions of behaviour. This insight disentangles the influences of salience on replay, suggesting that reinforcement learning is tuned by post-learning replay biased by reward-prediction error, not by reward per se. This work therefore provides a behavioural and theoretical toolkit with which to measure and interpret replay in striatal, hippocampal and neocortical circuits.

Emotion prediction errors guide socially adaptive behavior

2021 ◽  
Author(s):  
Joseph Heffner ◽  
Jae-Young Son ◽  
Oriel FeldmanHall
Keyword(s):  
Decision Making ◽  
Real Time ◽  
Adaptive Behavior ◽  
Emotional Response ◽  
New Method ◽  
Prediction Errors ◽  
Emotional Experiences ◽  
Past Work ◽  
Reward Prediction ◽  
Expected Outcomes

People make decisions based on deviations from expected outcomes, known as prediction errors. Past work has focused on reward prediction errors, largely ignoring violations of expected emotional experiences—emotion prediction errors. We leverage a new method to measure real-time fluctuations in emotion as people decide to punish or forgive others. Across four studies (N=1,016), we reveal that emotion and reward prediction errors have distinguishable contributions to choice, such that emotion prediction errors exert the strongest impact during decision-making. We additionally find that a choice to punish or forgive can be decoded in less than a second from an evolving emotional response, suggesting emotions swiftly influence choice. Finally, individuals reporting significant levels of depression exhibit selective impairments in using emotion—but not reward—prediction errors. Evidence for emotion prediction errors potently guiding social behaviors challenge standard decision-making models that have focused solely on reward.

scholarly journals Prefrontal solution to the bias-variance tradeoff during reinforcement learning

2020 ◽  
Author(s):  
Dongjae Kim ◽  
Jaeseung Jeong ◽  
Sang Wan Lee
Keyword(s):  
Adaptive Control ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Brain Regions ◽  
Decision Task ◽  
Prediction Errors ◽  
Model Based ◽  
Model Free ◽  
Bias Variance ◽  
The Brain

AbstractThe goal of learning is to maximize future rewards by minimizing prediction errors. Evidence have shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning.One sentence summaryA theoretical, behavioral, computational, and neural account of how the brain resolves the bias-variance tradeoff during reinforcement learning is described.

scholarly journals Attenuated directed exploration during reinforcement learning in gambling disorder

2019 ◽  
Author(s):  
A. Wiehler ◽  
K. Chakroun ◽  
J. Peters
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Gambling Disorder ◽  
Brain Activity ◽  
Clinical Status ◽  
Classical Problem ◽  
Behavioral Flexibility ◽  
Network Connectivity ◽  
Prediction Errors ◽  
Reward Contingencies

AbstractGambling disorder is a behavioral addiction associated with impairments in decision-making and reduced behavioral flexibility. Decision-making in volatile environments requires a flexible trade-off between exploitation of options with high expected values and exploration of novel options to adapt to changing reward contingencies. This classical problem is known as the exploration-exploitation dilemma. We hypothesized gambling disorder to be associated with a specific reduction in directed (uncertainty-based) exploration compared to healthy controls, accompanied by changes in brain activity in a fronto-parietal exploration-related network.Twenty-three frequent gamblers and nineteen matched controls performed a classical four-armed bandit task during functional magnetic resonance imaging. Computational modeling revealed that choice behavior in both groups contained signatures of directed exploration, random exploration and perseveration. Gamblers showed a specific reduction in directed exploration, while random exploration and perseveration were similar between groups.Neuroimaging revealed no evidence for group differences in neural representations of expected value and reward prediction errors. Likewise, our hypothesis of attenuated fronto-parietal exploration effects in gambling disorder was not supported. However, during directed exploration, gamblers showed reduced parietal and substantia nigra / ventral tegmental area activity. Cross-validated classification analyses revealed that connectivity in an exploration-related network was predictive of clinical status, suggesting alterations in network dynamics in gambling disorder.In sum, we show that reduced flexibility during reinforcement learning in volatile environments in gamblers is attributable to a reduction in directed exploration rather than an increase in perseveration. Neuroimaging findings suggest that patterns of network connectivity might be more diagnostic of gambling disorder than univariate value and prediction error effects. We provide a computational account of flexibility impairments in gamblers during reinforcement learning that might arise as a consequence of dopaminergic dysregulation in this disorder.

A further assessment of decision-making in anorexia nervosa

2015 ◽  
Vol 30 (1) ◽  
pp. 121-127 ◽  
Author(s):  
C. Adoue ◽  
I. Jaussent ◽  
E. Olié ◽  
S. Beziat ◽  
F. Van den Eynde ◽  
...  
Keyword(s):  
Decision Making ◽  
Major Depressive Disorder ◽  
Anorexia Nervosa ◽  
Depressive Disorder ◽  
Cognitive Processes ◽  
Depressive Disorders ◽  
Iowa Gambling Task ◽  
Learning Task ◽  
Adult Women ◽  
Major Depressive

AbstractObjective:Anorexia nervosa (AN) may be associated with impaired decision-making. Cognitive processes underlying this impairment remain unclear, mainly because previous assessments of this complex cognitive function were completed with a single test. Furthermore, clinical features such as mood status may impact this association. We aim to further explore the hypothesis of altered decision-making in AN.Method:Sixty-three adult women with AN and 49 female controls completed a clinical assessment and were assessed by three tasks related to decision-making [Iowa Gambling Task (IGT), Balloon Analogue Risk Task (BART), Probabilistic Reversal Learning Task (PRLT)].Results:People with AN had poorer performance on the IGT and made less risky choices on the BART, whereas performances were not different on PRLT. Notably, AN patients with a current major depressive disorder showed similar performance to those with no current major depressive disorder.Conclusion:These results tend to confirm an impaired decision making-process in people with AN and suggest that various cognitive processes such as inhibition to risk-taking or intolerance of uncertainty may underlie this condition Furthermore, these impairments seem unrelated to the potential co-occurent major depressive disorders.

scholarly journals Positive reward prediction errors during decision-making strengthen memory encoding

2019 ◽  
Vol 3 (7) ◽  
pp. 719-732 ◽  
Author(s):  
Anthony I. Jang ◽  
Matthew R. Nassar ◽  
Daniel G. Dillon ◽  
Michael J. Frank
Keyword(s):  
Decision Making ◽  
Prediction Errors ◽  
Memory Encoding ◽  
Reward Prediction
Sign in / Sign up

Export Citation Format

Share Document