Coding of the long-term value of multiple future rewards in the primate striatum

2013 ◽  
Vol 109 (4) ◽  
pp. 1140-1151 ◽  
Author(s):  
Hiroshi Yamada ◽  
Hitoshi Inokawa ◽  
Naoyuki Matsumoto ◽  
Yasumasa Ueda ◽  
Kazuki Enomoto ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Learning Theories ◽  
Projection Neurons ◽  
Striatal Neurons ◽  
Striatal Projection Neurons ◽  
Positive Correlations ◽  
Task Trial ◽  
The Cost ◽  
Near Future

Decisions maximizing benefits involve a tradeoff between the quantity of a reward and the cost of elapsed time until an animal receives it. The estimation of long-term reward values is critical to attain the most desirable outcomes over a certain period of time. Reinforcement learning theories have established algorithms to estimate the long-term reward values of multiple future rewards in which the values of future rewards are discounted as a function of how many steps of choices are necessary to achieve them. Here, we report that presumed striatal projection neurons represent the long-term values of multiple future rewards estimated by a standard reinforcement learning model while monkeys are engaged in a series of trial-and-error choices and adaptive decisions for multiple rewards. We found that the magnitude of activity of a subset of neurons was positively correlated with the long-term reward values, and that of another subset of neurons was negatively correlated throughout the entire decision-making process in individual trials: from the start of the task trial, estimation of the values and their comparison among alternatives, choice execution, and evaluation of the received rewards. An idiosyncratic finding was that neurons showing negative correlations represented reward values in the near future (high discounting), while neurons showing positive correlations represented reward values not only in the near future, but also in the far future (low discounting). These findings provide a new insight that long-term value signals are embedded in two subsets of striatal neurons as high and low discounting of multiple future rewards.

scholarly journals The Tortoise and the Hare: Interactions between Reinforcement Learning and Working Memory

2018 ◽  
Vol 30 (10) ◽  
pp. 1422-1432 ◽  
Author(s):  
Anne G. E. Collins
Keyword(s):  
Working Memory ◽  
Reinforcement Learning ◽  
Computational Modeling ◽  
Behavioral Experiment ◽  
Prediction Errors ◽  
Multiple Systems ◽  
Memory Interference ◽  
Neural Representations ◽  
The Cost

Learning to make rewarding choices in response to stimuli depends on a slow but steady process, reinforcement learning, and a fast and flexible, but capacity-limited process, working memory. Using both systems in parallel, with their contributions weighted based on performance, should allow us to leverage the best of each system: rapid early learning, supplemented by long-term robust acquisition. However, this assumes that using one process does not interfere with the other. We use computational modeling to investigate the interactions between the two processes in a behavioral experiment and show that working memory interferes with reinforcement learning. Previous research showed that neural representations of reward prediction errors, a key marker of reinforcement learning, were blunted when working memory was used for learning. We thus predicted that arbitrating in favor of working memory to learn faster in simple problems would weaken the reinforcement learning process. We tested this by measuring performance in a delayed testing phase where the use of working memory was impossible, and thus participant choices depended on reinforcement learning. Counterintuitively, but confirming our predictions, we observed that associations learned most easily were retained worse than associations learned slower: Using working memory to learn quickly came at the cost of long-term retention. Computational modeling confirmed that this could only be accounted for by working memory interference in reinforcement learning computations. These results further our understanding of how multiple systems contribute in parallel to human learning and may have important applications for education and computational psychiatry.

scholarly journals KV7 Channels Regulate Firing during Synaptic Integration in GABAergic Striatal Neurons

2015 ◽  
Vol 2015 ◽  
pp. 1-18 ◽  
Author(s):  
M. Belén Pérez-Ramírez ◽  
Antonio Laville ◽  
Dagoberto Tapia ◽  
Mariana Duhne ◽  
Esther Lara-González ◽  
...  
Keyword(s):  
Muscarinic Receptors ◽  
Synaptic Integration ◽  
Projection Neurons ◽  
Striatal Neurons ◽  
Kv7 Channels ◽  
Striatal Projection Neurons ◽  
Voltage Dependent ◽  
Cognitive Information ◽  
Firing Properties ◽  
The Impact

Striatal projection neurons (SPNs) process motor and cognitive information. Their activity is affected by Parkinson’s disease, in which dopamine concentration is decreased and acetylcholine concentration is increased. Acetylcholine activates muscarinic receptors in SPNs. Its main source is the cholinergic interneuron that responds with a briefer latency than SPNs during a cortical command. Therefore, an important question is whether muscarinic G-protein coupled receptors and their signaling cascades are fast enough to intervene during synaptic responses to regulate synaptic integration and firing. One of the most known voltage dependent channels regulated by muscarinic receptors is theKV7/KCNQ channel. It is not known whether these channels regulate the integration of suprathreshold corticostriatal responses. Here, we study the impact of cholinergic muscarinic modulation on the synaptic response of SPNs by regulatingKV7 channels. We found thatKV7 channels regulate corticostriatal synaptic integration and that this modulation occurs in the dendritic/spines compartment. In contrast, it is negligible in the somatic compartment. This modulation occurs on sub- and suprathreshold responses and lasts during the whole duration of the responses, hundreds of milliseconds, greatly altering SPNs firing properties. This modulation affected the behavior of the striatal microcircuit.

scholarly journals Repression of Dlx1/2 Signaling by Nolz-1/Znf503 is Essential for Parcellation of the Striatal Complex into Dorsal and Ventral Striatum

2018 ◽  
Author(s):  
Kuan-Ming Lu ◽  
Shih-Yun Chen ◽  
Hsin-An Ko ◽  
Ting-Hao Huang ◽  
Janice Hsin-Jou Hao ◽  
...  
Keyword(s):  
Transcription Factors ◽  
Ventral Striatum ◽  
Dorsal Striatum ◽  
Clinical Importance ◽  
Projection Neurons ◽  
Striatal Neurons ◽  
Addictive Disorders ◽  
Striatal Projection Neurons ◽  
Cells Of Origin ◽  
Set Up

ABSTRACTThe division of the striatum into dorsal and ventral districts is of central clinical importance. The dorsal striatum is differentially affected in Huntington’s disease, dopamine in the ventral striatum is differentially spared in Parkinson’s disease, and human brain imaging studies implicate the ventral striatum in addictive disorders. If fits that the dorsal striatum contains the cells of origin of the direct and indirect basal ganglia pathways for motor control. The ventral striatum is a node in neural circuits related to motivation and affect. Despite these striking neurobiologic contrasts, there is almost no information about how the dorsal and ventral divisions of the striatum are set up during development. Here, we demonstrate that interactions between the two key transcription factors Nolz-1 and Dlx1/2 control the migratory paths of developing striatal neurons to the dorsal or ventral striatum. Moreover, these same transcription factors control the cell identity of striatal projection neurons in both the dorsal and ventral striatum including the cell origin of the direct and indirect pathways. We show that Nolz-1 suppresses Dlx1/2 expression. Deletion of Nolz-1 or over-expression of Dlx1/2 can produce a striatal phenotype characterized by withered dorsal striatum and a swollen ventral striatum, and that we can rescue this phenotype by manipulating the interactions between Nolz-1 and Dlx1/2 transcription factors. This evidence suggests that the fundamental basis for divisions of the striatum known to be differentially vulnerable at maturity is already encoded by the time embryonic striatal neurons begin their migrations into the developing striatum.

scholarly journals Dopamine D4 Receptor Is a Regulator of Morphine-Induced Plasticity in the Rat Dorsal Striatum

Cells ◽  
2021 ◽  
Vol 11 (1) ◽  
pp. 31
Author(s):  
Alicia Rivera ◽  
Diana Suárez-Boomgaard ◽  
Cristina Miguelez ◽  
Alejandra Valderrama-Carvajal ◽  
Jérôme Baufreton ◽  
...  
Keyword(s):  
Critical Role ◽  
Combined Treatment ◽  
Dorsal Striatum ◽  
Prolonged Treatment ◽  
Projection Neurons ◽  
Spine Density ◽  
Dopamine D4 Receptor ◽  
Striatal Projection Neurons ◽  
D4 Receptor

Long-term exposition to morphine elicits structural and synaptic plasticity in reward-related regions of the brain, playing a critical role in addiction. However, morphine-induced neuroadaptations in the dorsal striatum have been poorly studied despite its key function in drug-related habit learning. Here, we show that prolonged treatment with morphine triggered the retraction of the dendritic arbor and the loss of dendritic spines in the dorsal striatal projection neurons (MSNs). In an attempt to extend previous findings, we also explored whether the dopamine D4 receptor (D4R) could modulate striatal morphine-induced plasticity. The combined treatment of morphine with the D4R agonist PD168,077 produced an expansion of the MSNs dendritic arbors and restored dendritic spine density. At the electrophysiological level, PD168,077 in combination with morphine altered the electrical properties of the MSNs and decreased their excitability. Finally, results from the sustantia nigra showed that PD168,077 counteracted morphine-induced upregulation of μ opioid receptors (MOR) in striatonigral projections and downregulation of G protein-gated inward rectifier K+ channels (GIRK1 and GIRK2) in dopaminergic cells. The present results highlight the key function of D4R modulating morphine-induced plasticity in the dorsal striatum. Thus, D4R could represent a valuable pharmacological target for the safety use of morphine in pain management.

scholarly journals The tortoise and the hare: interactions between reinforcement learning and working memory

2017 ◽  
Author(s):  
Anne G.E. Collins
Keyword(s):  
Working Memory ◽  
Reinforcement Learning ◽  
Computational Modeling ◽  
Behavioral Experiment ◽  
Prediction Errors ◽  
Multiple Systems ◽  
Memory Interference ◽  
Neural Representations ◽  
The Cost

AbstractLearning to make rewarding choices in response to stimuli depends on a slow but steady process, reinforcement learning, and a fast and flexible, but capacity limited process, working memory. Using both systems in parallel, with their contributions weighted based on performance, should allow us to leverage the best of each system: rapid early learning, supplemented by long term robust acquisition. However, this assumes that using one process does not interfere with the other. We use computational modeling to investigate the interactions between the two processes in a behavioral experiment, and show that working memory interferes with reinforcement learning. Previous research showed that neural representations of reward prediction errors, a key marker of reinforcement learning, were blunted when working memory was used for learning. We thus predicted that arbitrating in favor of working memory to learn faster in simple problems would weaken the reinforcement learning process. We tested this by measuring performance in a delayed testing phase where the use of working memory was impossible, and thus subject choices depended on reinforcement learning. Counter-intuitively, but confirming our predictions, we observed that associations learned most easily were retained worse than associations learned slower: using working memory to learn quickly came at the cost of long-term retention. Computational modeling confirmed that this could only be accounted for by working memory interference in reinforcement learning computations. These results further our understanding of how multiple systems contribute in parallel to human learning, and may have important applications for education and computational psychiatry.

scholarly journals Coincidence of cholinergic pauses, dopaminergic activation and depolarization drives synaptic plasticity in the striatum

2019 ◽  
Author(s):  
Yan-Feng Zhang ◽  
Simon D. Fisher ◽  
Manfred Oswald ◽  
Jeffery R. Wickens ◽  
John N. J. Reynolds
Keyword(s):  
Synaptic Plasticity ◽  
Reinforcement Learning ◽  
Long Term Potentiation ◽  
Dopamine Neurons ◽  
Projection Neurons ◽  
Cholinergic Interneurons ◽  
Term Potentiation ◽  
Phasic Activation

AbstractPauses in the firing of tonically-active cholinergic interneurons (ChIs) in the striatum coincide with phasic activation of dopamine neurons during reinforcement learning. However, how this pause influences cellular substrates of learning is unclear. Using two in vivo paradigms, we report that long-term potentiation (LTP) at corticostriatal synapses with spiny projection neurons (SPNs) is dependent on the temporal coincidence of ChI pause and dopamine phasic activation, critically accompanied by SPN depolarization.

scholarly journals Parcellation of the striatal complex into dorsal and ventral districts

2020 ◽  
Vol 117 (13) ◽  
pp. 7418-7429 ◽  
Author(s):  
Shih-Yun Chen ◽  
Kuan-Ming Lu ◽  
Hsin-An Ko ◽  
Ting-Hao Huang ◽  
Janice Hsin-Jou Hao ◽  
...  
Keyword(s):  
Transcription Factors ◽  
Ventral Striatum ◽  
Dorsal Striatum ◽  
Clinical Importance ◽  
Projection Neurons ◽  
Striatal Neurons ◽  
Striatal Projection Neurons ◽  
Tier System ◽  
And Migration ◽  
Set Up

The striatal complex of basal ganglia comprises two functionally distinct districts. The dorsal district controls motor and cognitive functions. The ventral district regulates the limbic function of motivation, reward, and emotion. The dorsoventral parcellation of the striatum also is of clinical importance as differential striatal pathophysiologies occur in Huntington’s disease, Parkinson’s disease, and drug addiction disorders. Despite these striking neurobiologic contrasts, it is largely unknown how the dorsal and ventral divisions of the striatum are set up. Here, we demonstrate that interactions between the two key transcription factors Nolz-1 and Dlx1/2 control the migratory paths of striatal neurons to the dorsal or ventral striatum. Moreover, these same transcription factors control the cell identity of striatal projection neurons in both the dorsal and the ventral striata including the D1-direct and D2-indirect pathways. We show that Nolz-1, through the I12b enhancer, represses Dlx1/2, allowing normal migration of striatal neurons to dorsal and ventral locations. We demonstrate that deletion, up-regulation, and down-regulation of Nolz-1 and Dlx1/2 can produce a striatal phenotype characterized by a withered dorsal striatum and an enlarged ventral striatum and that we can rescue this phenotype by manipulating the interactions between Nolz-1 and Dlx1/2 transcription factors. Our study indicates that the two-tier system of striatal complex is built by coupling of cell-type identity and migration and suggests that the fundamental basis for divisions of the striatum known to be differentially vulnerable at maturity is already encoded by the time embryonic striatal neurons begin their migrations into developing striata.

scholarly journals Role of striatal ΔFosB in l-Dopa–induced dyskinesias of parkinsonian nonhuman primates

2019 ◽  
Vol 116 (37) ◽  
pp. 18664-18672 ◽  
Author(s):  
Goichi Beck ◽  
Arun Singh ◽  
Jie Zhang ◽  
Lisa F. Potts ◽  
Jong-Min Woo ◽  
...  
Keyword(s):  
Motor Response ◽  
Nonhuman Primates ◽  
Firing Frequency ◽  
Projection Neurons ◽  
Neuronal Responses ◽  
Motor Responses ◽  
Striatal Projection Neurons ◽  
Adaptive Mechanisms

Long-term dopamine (DA) replacement therapy in Parkinson’s disease (PD) leads to the development of abnormal involuntary movements known asl-Dopa–induced dyskinesia (LID). The transcription factor ΔFosB that is highly up-regulated in the striatum following chronicl-Dopa exposure may participate in the mechanisms of altered neuronal responses to DA generating LID. To identify intrinsic effects of elevated ΔFosB onl-Dopa responses, we induced transgenic ΔFosB overexpression in the striatum of parkinsonian nonhuman primates kept naïve ofl-Dopa treatment. Elevated ΔFosB levels led to consistent appearance of LID since the initial acutel-Dopa tests. In line with this motor response, striatal projection neurons (SPNs) responded to DA with changes in firing frequency that reversed at the peak of the motor response, and these unstable SPN activity changes in response to DA are typically associated with the emergence of LID. Transgenic ΔFosB overexpression also induced up-regulation of other molecular markers of LID. These results support an autonomous role of striatal ΔFosB in the adaptive mechanisms altering motor responses to chronic DA replacement in PD.

Sign in / Sign up

Export Citation Format

Share Document