Reward activity in ventral pallidum tracks satiety-sensitive preference and drives choice behavior

A key function of the nervous system is producing adaptive behavior across changing conditions, like physiological state. Although states like thirst and hunger are known to impact decision-making, the neurobiology of this phenomenon has been studied minimally. Here, we tracked evolving preference for sucrose and water as rats proceeded from a thirsty to sated state. As rats shifted from water choices to sucrose choices across the session, the activity of a majority of neurons in the ventral pallidum, a region crucial for reward-related behaviors, closely matched the evolving behavioral preference. The timing of this signal followed the pattern of a reward prediction error, occurring at the cue or the reward depending on when reward identity was revealed. Additionally, optogenetic stimulation of ventral pallidum neurons at the time of reward was able to reverse behavioral preference. Our results suggest that ventral pallidum neurons guide reward-related decisions across changing physiological states.

Download Full-text

Disruptions in effort-based decision-making following acute optogenetic stimulation of ventral tegmental area dopamine cells

Learning & Memory ◽

10.1101/lm.053082.120 ◽

2021 ◽

Vol 28 (4) ◽

pp. 104-108

Author(s):

Benjamin R. Fry ◽

Nathan T. Pence ◽

Andrew McLocklin ◽

Alexander W. Johnson

Keyword(s):

Decision Making ◽

Ventral Tegmental Area ◽

Optogenetic Stimulation ◽

Ventral Tegmental ◽

Stimulation Of

Download Full-text

Disruptions in effort-based decision-making following acute optogenetic stimulation of ventral tegmental area dopamine cells

10.1101/2020.12.09.417832 ◽

2020 ◽

Author(s):

Benjamin R. Fry ◽

Nathan T. Pence ◽

Andrew McLocklin ◽

Alexander W. Johnson

Keyword(s):

Decision Making ◽

Tyrosine Hydroxylase ◽

Ventral Tegmental Area ◽

Dopamine System ◽

Laser Stimulation ◽

Optogenetic Stimulation ◽

Ventral Tegmental ◽

Stimulation Of

AbstractThe dopamine system has been implicated in decision-making particularly when associated with effortful behavior. We examined acute optogenetic stimulation of dopamine cells in the ventral tegmental area (VTA) as mice engaged in an effort-based decision-making task. Tyrosine hydroxylase-Cre mice were injected with Cre-dependent ChR2 or control eYFP in VTA. While eYFP control mice showed effortful discounting, laser stimulation of dopamine cells in ChR2 animals disrupted effort-based decision-making by reducing choice towards the lever associated with a preferred outcome and greater effort, without affecting discrimination processes or nonspecific motoric behaviors. These findings suggest increases in dopamine activity can disrupt effort-based decision-making.

Download Full-text

A quantitative reward prediction error signal in the ventral pallidum

Nature Neuroscience ◽

10.1038/s41593-020-0688-5 ◽

2020 ◽

Vol 23 (10) ◽

pp. 1267-1276 ◽

Cited By ~ 2

Author(s):

David J. Ottenheimer ◽

Bilal A. Bari ◽

Elissa Sutlief ◽

Kurt M. Fraser ◽

Tabitha H. Kim ◽

...

Keyword(s):

Prediction Error ◽

Ventral Pallidum ◽

Error Signal ◽

Reward Prediction Error ◽

Reward Prediction

Download Full-text

Novelty is not Surprise: Exploration and learning in human sequential decision-making

10.1101/2020.09.24.311084 ◽

2020 ◽

Author(s):

He A. Xu ◽

Alireza Modirshanechi ◽

Marco P. Lehmann ◽

Wulfram Gerstner ◽

Michael H. Herzog

Keyword(s):

Decision Making ◽

Prediction Error ◽

Event Related Potential ◽

Sequential Decision Making ◽

Sequential Decision ◽

Model Free ◽

Reward Prediction ◽

Rate Of Learning ◽

Human Participants ◽

Exploration Exploitation

AbstractDrivers of reinforcement learning (RL), beyond reward, are controversially debated. Novelty and surprise are often used equivocally in this debate. Here, using a deep sequential decision-making paradigm, we show that reward, novelty, and surprise play different roles in human RL. Surprise controls the rate of learning, whereas novelty and the novelty prediction error (NPE) drive exploration. Exploitation is dominated by model-free (habitual) action choices. A theory that takes these separate effects into account predicts on average 73 percent of the action choices of human participants after the first encounter of a reward and allows us to dissociate surprise and novelty in the EEG signal. While the event-related potential (ERP) at around 300ms is positively correlated with surprise, novelty, NPE, reward, and the reward prediction error, the ERP response to novelty and NPE starts earlier than that to surprise.

Download Full-text

Aversion encoded in the subthalamic nucleus

10.1101/2020.07.09.195610 ◽

2020 ◽

Author(s):

Gian Pietro Serra ◽

Adriane Guillaumin ◽

Jérome Baufreton ◽

François Georges ◽

Åsa Wallén-Mackenzie

Keyword(s):

Basal Ganglia ◽

Subthalamic Nucleus ◽

Place Preference ◽

Ventral Pallidum ◽

Lateral Habenula ◽

Place Aversion ◽

Optogenetic Stimulation ◽

Ongoing Behavior ◽

Aversive Behavior ◽

Stimulation Of

AbstractActivation of the subthalamic nucleus (STN) is associated with the stopping of ongoing behavior via the basal ganglia. However, we recently observed that optogenetic STN excitation induced a strong jumping/escaping behavior. We hypothesized that STN activation is aversive. To test this, place preference was assessed. Optogenetic excitation of the STN caused potent place aversion. Causality between STN activation and aversion has not been demonstrated previously. The lateral habenula (LHb) is a critical hub for aversion. Optogenetic stimulation of the STN indeed caused firing of LHb neurons, but with delay, suggesting the involvement of a polysynaptic circuit. To unravel a putative pathway, the ventral pallidum (VP) was investigated. VP receives projections from the STN and in turn projects to the LHb. Optogenetic excitation of STN-VP terminals caused firing of VP neurons and induced aversive behavior. This study identifies the STN as critical hub for aversion, potentially mediated via an STN-VP-LHb pathway.

Download Full-text

Impaired expected value computations coupled with overreliance on prediction error learning in schizophrenia

10.1101/238089 ◽

2017 ◽

Cited By ~ 1

Author(s):

D Hernaus ◽

JM Gold ◽

JA Waltz ◽

MJ Frank

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Prediction Error ◽

Expected Value ◽

Strongly Correlated ◽

Transfer Phase ◽

Robust Performance ◽

Reward Prediction ◽

Error Learning ◽

Choice Bias

AbstractBackgroundWhile many have emphasized impaired reward prediction error (RPE) signaling in schizophrenia, multiple studies suggest that some decision-making deficits may arise from overreliance on RPE systems together with a compromised ability to represent expected value. Guided by computational frameworks, we formulated and tested two scenarios in which maladaptive representation of expected value should be most evident, thereby delineating conditions that may evoke decision-making impairments in schizophrenia.MethodsIn a modified reinforcement learning paradigm, 42 medicated people with schizophrenia (PSZ) and 36 healthy volunteers learned to select the most frequently rewarded option in a 75-25 pair: once when presented with more deterministic (90–10) and once when presented with more probabilistic (60–40) pairs. Novel and old combinations of choice options were presented in a subsequent transfer phase. Computational modeling was employed to elucidate contributions from RPE systems (“actor-critic”) and expected value (“Q-leaming”).ResultsPSZ showed robust performance impairments with increasing value difference between two competing options, which strongly correlated with decreased contributions from expected value-based (“Q-leaming”) learning. Moreover, a subtle yet consistent contextual choice bias for the “probabilistic” 75 option was present in PSZ, which could be accounted for by a context-dependent RPE in the “actor-critic”.ConclusionsWe provide evidence that decision-making impairments in schizophrenia increase monotonically with demands placed on expected value computations. A contextual choice bias is consistent with overreliance on RPE-based learning, which may signify a deficit secondary to the maladaptive representation of expected value. These results shed new light on conditions under which decisionmaking impairments may arise.

Download Full-text

Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development

NeuroImage ◽

10.1016/j.neuroimage.2014.09.018 ◽

2015 ◽

Vol 104 ◽

pp. 347-354 ◽

Cited By ~ 70

Author(s):

Tobias U. Hauser ◽

Reto Iannaccone ◽

Susanne Walitza ◽

Daniel Brandeis ◽

Silvia Brem

Keyword(s):

Decision Making ◽

Cognitive Flexibility ◽

Prediction Error ◽

Error Processing ◽

Reward Prediction Error ◽

Reward Prediction ◽

Behavioral Mechanisms ◽

Adaptive Decision Making

Download Full-text

The contribution of striatal pseudo-reward prediction errors to value-based decision-making

10.1101/097873 ◽

2017 ◽

Author(s):

Ernest Mas-Herrero ◽

Guillaume Sescousse ◽

Roshan Cools ◽

Josep Marco-Pallarés

Keyword(s):

Decision Making ◽

Choice Behavior ◽

Prediction Errors ◽

Life Outcomes ◽

Stimulus Response ◽

Hierarchical Reinforcement Learning ◽

Reward Prediction ◽

Fmri Study ◽

Reward Contingencies ◽

The Brain

AbstractMost studies that have investigated the brain mechanisms underlying learning have focused on the ability to learn simple stimulus-response associations. However, in everyday life, outcomes are often obtained through complex behavioral patterns involving a series of actions. In such scenarios, parallel learning systems are important to reduce the complexity of the learning problem, as proposed in the framework of hierarchical reinforcement learning (HRL). One of the key features of HRL is the computation of pseudo-reward prediction errors (PRPEs) which allow the reinforcement of actions that led to a sub-goal before the final goal itself is achieved. Here we wanted to test the hypothesis that, despite not carrying any rewarding value per se, pseudo-rewards might generate a bias in choice behavior when reward contingencies are not well-known or uncertain. Second, we also hypothesized that this bias might be related to the strength of PRPE striatal representations. In order to test these ideas, we developed a novel decision-making paradigm to assess reward prediction errors (RPEs) and PRPEs in two studies (fMRI study: n = 20; behavioural study: n = 19). Our results show that overall participants developed a preference for the most pseudo-rewarding option throughout the task, even though it did not lead to more monetary rewards. fMRI analyses revealed that this preference was predicted by individual differences in the relative striatal sensitivity to PRPEs vs RPEs. Together, our results indicate that pseudo-rewards generate learning signals in the striatum and subsequently bias choice behavior despite their lack of association with actual reward.

Download Full-text