scholarly journals Diffusion modeling reveals reinforcement learning impairments in gambling disorder that are linked to attenuated ventromedial prefrontal cortex value representations

2020 ◽  
Author(s):  
Antonius Wiehler ◽  
Jan Peters
Keyword(s):  
Prefrontal Cortex ◽  
Reinforcement Learning ◽  
Gambling Disorder ◽  
Ventromedial Prefrontal Cortex ◽  
Response Threshold ◽  
Boundary Separation ◽  
Learning Impairments ◽  
Model Based ◽  
Learning Rates ◽  
Over Time

AbstractGambling disorder is associated with deficits in classical feedback-based learning tasks, but the computational mechanisms underlying such learning impairments are still poorly understood. Here, we examined this question using a combination of computational modeling and functional resonance imaging (fMRI) in gambling disorder participants (n=23) and matched controls (n=19). Participants performed a simple reinforcement learning task with two pairs of stimuli (80% vs. 20% reinforcement rates per pair). As predicted, gamblers made significantly fewer selections of the optimal stimulus, while overall response times (RTs) were not significantly different between groups. We then used comprehensive modeling using reinforcement learning drift diffusion models (RLDDMs) in combination with hierarchical Bayesian parameter estimation to shed light on the computational underpinnings of this performance impairment. In both groups, an RLDDM in which both non-decision time and response threshold (boundary separation) changed over the course of the experiment accounted for the data best. The model showed good parameter recovery, and posterior predictive checks revealed that in both groups, the model reproduced the evolution of both accuracy and RTs over time. Examination of the group-wise posterior distributions revealed that the learning impairment in gamblers was attributable to both reduced learning rates and a more rapid reduction in boundary separation over time, compared to controls. Furthermore, gamblers also showed substantially shorter non-decision times. Model-based imaging analyses then revealed that value representations in gamblers in the ventromedial prefrontal cortex were attenuated compared to controls, and these effects were partly associated with model-based learning rates. Exploratory analyses revealed that a more anterior ventromedial prefrontal cortex cluster showed attenuations in value representations in proportion to gambling disorder severity in gamblers. Taken together, our findings reveal computational mechanisms underlying reinforcement learning impairments in gambling disorder, and confirm the ventromedial prefrontal cortex and as a critical neural hub in this disorder.

scholarly journals Neurostimulation Reveals Context-Dependent Arbitration Between Model-Based and Model-Free Reinforcement Learning

2019 ◽  
Vol 29 (11) ◽  
pp. 4850-4862 ◽  
Author(s):  
Sebastian Weissengruber ◽  
Sang Wan Lee ◽  
John P O’Doherty ◽  
Christian C Ruff
Keyword(s):  
Reinforcement Learning ◽  
Neural Mechanism ◽  
Ventrolateral Prefrontal Cortex ◽  
Control Behavior ◽  
Model Based ◽  
Model Free ◽  
Context Dependent ◽  
The Brain ◽  
Over Time ◽  
Environmental Demands

Abstract While it is established that humans use model-based (MB) and model-free (MF) reinforcement learning in a complementary fashion, much less is known about how the brain determines which of these systems should control behavior at any given moment. Here we provide causal evidence for a neural mechanism that acts as a context-dependent arbitrator between both systems. We applied excitatory and inhibitory transcranial direct current stimulation over a region of the left ventrolateral prefrontal cortex previously found to encode the reliability of both learning systems. The opposing neural interventions resulted in a bidirectional shift of control between MB and MF learning. Stimulation also affected the sensitivity of the arbitration mechanism itself, as it changed how often subjects switched between the dominant system over time. Both of these effects depended on varying task contexts that either favored MB or MF control, indicating that this arbitration mechanism is not context-invariant but flexibly incorporates information about current environmental demands.

scholarly journals Value generalization in human avoidance learning

2017 ◽  
Author(s):  
Agnes Norbury ◽  
Trevor W. Robbins ◽  
Ben Seymour
Keyword(s):  
Decision Making ◽  
Prefrontal Cortex ◽  
Reinforcement Learning ◽  
Avoidance Learning ◽  
Anterior Insula ◽  
Ventromedial Prefrontal Cortex ◽  
Sensory Cortex ◽  
Primary Sensory Cortex ◽  
Human Volunteers ◽  
Mediate Generalization

SummaryGeneralization during aversive decision-making allows us to avoid a broad range of potential threats following experience with a limited set of exemplars. However, over-generalization, resulting in excessive and inappropriate avoidance, has been implicated in a variety of psychological disorders. Here, we use reinforcement learning modelling to dissect out different contributions to the generalization of instrumental avoidance in two groups of human volunteers (N=26, N=482). We found that generalization of avoidance could be parsed into perceptual and value-based processes, and further, that value-based generalization could be subdivided into that relating to aversive and neutral feedback - with corresponding circuits including primary sensory cortex, anterior insula, and ventromedial prefrontal cortex, respectively. Further, generalization from aversive, but not neutral, feedback was associated with self-reported anxiety and intrusive thoughts. These results reveal a set of distinct mechanisms that mediate generalization in avoidance learning, and show how specific individual differences within them can yield anxiety.

scholarly journals Increased ventromedial prefrontal cortex activity in adolescence benefits prosocial reinforcement learning

2021 ◽  
Author(s):  
Bianca Westhoff ◽  
Neeltje E. Blankenstein ◽  
Elisabeth Schreuders ◽  
Eveline A. Crone ◽  
Anna C. K. van Duijvenvoorde
Keyword(s):  
Prefrontal Cortex ◽  
Reinforcement Learning ◽  
Functional Neuroimaging ◽  
Learning Task ◽  
Developmental Trajectory ◽  
Ventromedial Prefrontal Cortex ◽  
Prediction Errors ◽  
Age Related ◽  
Probabilistic Reinforcement ◽  
Neuroimaging Study

AbstractLearning which of our behaviors benefit others contributes to social bonding and being liked by others. An important period for the development of (pro)social behavior is adolescence, in which peers become more salient and relationships intensify. It is, however, unknown how learning to benefit others develops across adolescence and what the underlying cognitive and neural mechanisms are. In this functional neuroimaging study, we assessed learning for self and others (i.e., prosocial learning) and the concurring neural tracking of prediction errors across adolescence (ages 9-21, N=74). Participants performed a two-choice probabilistic reinforcement learning task in which outcomes resulted in monetary consequences for themselves, an unknown other, or no one. Participants from all ages were able to learn for themselves and others, but learning for others showed a more protracted developmental trajectory. Prediction errors for self were observed in the ventral striatum and showed no age-related differences. However, prediction error coding for others was specifically observed in the ventromedial prefrontal cortex and showed age-related increases. These results reveal insights into the computational mechanisms of learning for others across adolescence, and highlight that learning for self and others show different age-related patterns.

scholarly journals Responses to heartbeats in ventromedial prefrontal cortex contribute to subjective preference-based decisions

2019 ◽  
Author(s):  
Damiano Azzalini ◽  
Anne Buot ◽  
Stefano Palminteri ◽  
Catherine Tallon-Baudry
Keyword(s):  
Prefrontal Cortex ◽  
Ventromedial Prefrontal Cortex ◽  
Functional Coupling ◽  
Self Reflection ◽  
Neural Interaction ◽  
Subjective Value ◽  
Choice Consistency ◽  
Neural Monitoring ◽  
Related Process ◽  
Over Time

AbstractForrest Gump or Matrix? Preference-based decisions are subjective and entail self-reflection. However, these self-related features are unaccounted for by known neural mechanisms of valuation and choice. Self-related processes have been linked to a basic interoceptive biological mechanism, the neural monitoring of heartbeats, in particular in ventromedial prefrontal cortex (vmPFC), a region also involved in value encoding. We thus hypothesized a functional coupling between the neural monitoring of heartbeats and the precision of value encoding in vmPFC. Human participants were presented with pairs of movie titles. They indicated either which movie they preferred, or performed a control objective visual discrimination that did not require self-reflection. Using magnetoencephalography, we measured heartbeat-evoked responses (HERs) before option presentation, and confirmed that HERs in vmPFC were larger when preparing to the subjective, self-related task. We retrieved the expected cortical value network during choice with time-resolved statistical modeling. Crucially, we show that larger HERs before option presentation are followed by stronger value encoding during choice in vmPFC. This effect is independent of the overall vmPFC baseline activity. The neural interaction between HERs and value encoding predicted preference-based choice consistency over time, accounting for both inter-individual differences and trial-to-trial fluctuations within individuals. Neither cardiac activity nor arousal fluctuations could account for any of the effects. HERs did not interact with the encoding of perceptual evidence in the discrimination task. Our results show that the self-reflection underlying preference-based decisions involves the integration of HERs to subjective value encoding in vmPFC, and that this integration contributes to preference stability.Significance statementDeciding whether you prefer Forrest Gump or Matrix is based on subjective values, which only you, the decision-maker, can estimate and compare, by asking yourself. Yet, how self-reflection is biologically implemented and its contribution to subjective valuation are not known. We show that in ventromedial prefrontal cortex, the neural response to heartbeats, an interoceptive self-related process, influences the cortical representation of subjective value. The neural interaction between the cortical monitoring of heartbeats and value encoding predicts choice consistency, i.e. whether you consistently prefer Forrest Gump over Matrix over time. Our results pave the way for the quantification of self-related process in decision making and may shed new light on the relationship between maladaptive decisions and impaired interoception.

scholarly journals Increased Ventromedial Prefrontal Cortex Activity in Adolescence Benefits Prosocial Reinforcement Learning

2021 ◽  
pp. 101018
Author(s):  
Bianca Westhoff ◽  
Neeltje E. Blankenstein ◽  
Elisabeth Schreuders ◽  
Eveline A. Crone ◽  
Anna C.K. van Duijvenvoorde
Keyword(s):  
Prefrontal Cortex ◽  
Reinforcement Learning ◽  
Ventromedial Prefrontal Cortex

scholarly journals Primate ventromedial prefrontal cortex neurons continuously encode the willingness to engage in reward directed behavior

2016 ◽  
Author(s):  
Aurore San Galli ◽  
Chiara Varazzani ◽  
Raphaelle Abitbol ◽  
Mathias Pessiglione ◽  
Sebastien Bouret
Keyword(s):  
Prefrontal Cortex ◽  
Visual Information ◽  
Contextual Information ◽  
Contextual Factors ◽  
Ventromedial Prefrontal Cortex ◽  
Complex Environment ◽  
Motivational States ◽  
Rapid Responses ◽  
Individual Task ◽  
Over Time

AbstractTo survive in their complex environment, primates must integrate information over time and adjust their actions beyond immediate events. The underlying neurobiological processes, however, remain unclear. Here, we assessed the contribution of the ventromedial prefrontal cortex (VMPFC), a brain region important for value-based decision making. We recorded single VMPFC neurons in monkeys performing a task where obtaining fluid rewards required squeezing a grip. The willingness to perform the action was modulated not only by visual information about Effort and Reward levels, but also by contextual factors such as Trial Number (i.e fatigue and/or satiety) or behavior in recent trials. A greater fraction of VMPFC neurons encoded contextual information, compared to visual stimuli. Moreover, the dynamics of VMPFC firing was more closely related to slow changes in motivational states driven by these contextual factors rather than rapid responses to individual task events. Thus, the firing of VMPFC neurons continuously integrated contextual information and reliably predicted the monkey’s willingness to perform the task. This function might be critical when animals forage in a complex environment and need to integrate information over time. Its relation with motivational states also resonates with the VMPFC implication in the default mode or in mood disorders.

scholarly journals Pure Correlates of Exploration and Exploitation in the Human Brain

2017 ◽  
Author(s):  
Tommy C. Blanchard ◽  
Samuel J. Gershman
Keyword(s):  
Prefrontal Cortex ◽  
Reinforcement Learning ◽  
Human Brain ◽  
Anterior Cingulate Cortex ◽  
Fundamental Problem ◽  
Cingulate Cortex ◽  
Ventromedial Prefrontal Cortex ◽  
Anterior Cingulate ◽  
Exploration And Exploitation ◽  
Exploration Exploitation

AbstractBalancing exploration and exploitation is a fundamental problem in reinforcement learning. Previous neuroimaging studies of the exploration-exploitation dilemma could not completely disentangle these two processes, making it difficult to unambiguously identify their neural signatures. We overcome this problem using a task in which subjects can either observe (pure exploration) or bet (pure exploitation). Insula and dorsal anterior cingulate cortex showed significantly greater activity on observe trials compared to bet trials, suggesting that these regions play a role in driving exploration. A model-based analysis of task performance suggested that subjects chose to observe until a critical evidence threshold was reached. We observed a neural signature of this evidence accumulation process in ventromedial prefrontal cortex. These findings support theories positing an important role for anterior cingulate cortex in exploration, while also providing a new perspective on the roles of insula and ventromedial prefrontal cortex.Significance StatementSitting down at a familiar restaurant, you may choose to order an old favorite or sample a new dish. In reinforcement learning theory, this is known as the exploration-exploitation dilemma. The optimal solution is known to be intractable; therefore, humans must use heuristic strategies. Behavioral studies have revealed several candidate strategies, but identifying the neural mechanisms underlying these strategies is complicated due to the fact that exploration and exploitation are not perfectly dissociable in standard tasks. Using an “observe or bet” task, we identify for the first time pure neural correlates of exploration and exploitation in the human brain.

scholarly journals Reduced model-based decision-making in gambling disorder

2019 ◽  
Vol 9 (1) ◽  
Author(s):  
Florent Wyckmans ◽  
A. Ross Otto ◽  
Miriam Sebold ◽  
Nathaniel Daw ◽  
Antoine Bechara ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Gambling Disorder ◽  
Reaction Times ◽  
Learning System ◽  
Behavioral Learning ◽  
Model Based ◽  
Model Free ◽  
Effective Interventions ◽  
Compulsive Behaviors

AbstractCompulsive behaviors (e.g., addiction) can be viewed as an aberrant decision process where inflexible reactions automatically evoked by stimuli (habit) take control over decision making to the detriment of a more flexible (goal-oriented) behavioral learning system. These behaviors are thought to arise from learning algorithms known as “model-based” and “model-free” reinforcement learning. Gambling disorder, a form of addiction without the confound of neurotoxic effects of drugs, showed impaired goal-directed control but the way in which problem gamblers (PG) orchestrate model-based and model-free strategies has not been evaluated. Forty-nine PG and 33 healthy participants (CP) completed a two-step sequential choice task for which model-based and model-free learning have distinct and identifiable trial-by-trial learning signatures. The influence of common psychopathological comorbidities on those two forms of learning were investigated. PG showed impaired model-based learning, particularly after unrewarded outcomes. In addition, PG exhibited faster reaction times than CP following unrewarded decisions. Troubled mood, higher impulsivity (i.e., positive and negative urgency) and current and chronic stress reported via questionnaires did not account for those results. These findings demonstrate specific reinforcement learning and decision-making deficits in behavioral addiction that advances our understanding and may be important dimensions for designing effective interventions.

Sign in / Sign up

Export Citation Format

Share Document