scholarly journals Adapting the flow of time with dopamine

2019 ◽  
Vol 121 (5) ◽  
pp. 1748-1760 ◽  
Author(s):  
John G. Mikhael ◽  
Samuel J. Gershman
Keyword(s):  
Reinforcement Learning ◽  
Opposite Effect ◽  
Large Body ◽  
Interval Timing ◽  
Striatal Neurons ◽  
Time Intervals ◽  
Learning Tasks ◽  
Flow Of Time ◽  
Open Question ◽  
Established Role

The modulation of interval timing by dopamine (DA) has been well established over decades of research. The nature of this modulation, however, has remained controversial: Although the pharmacological evidence has largely suggested that time intervals are overestimated with higher DA levels, more recent optogenetic work has shown the opposite effect. In addition, a large body of work has asserted DA’s role as a “reward prediction error” (RPE), or a teaching signal that allows the basal ganglia to learn to predict future rewards in reinforcement learning tasks. Whether these two seemingly disparate accounts of DA may be related has remained an open question. By taking a reinforcement learning-based approach to interval timing, we show here that the RPE interpretation of DA naturally extends to its role as a modulator of timekeeping and furthermore that this view reconciles the seemingly conflicting observations. We derive a biologically plausible, DA-dependent plasticity rule that can modulate the rate of timekeeping in either direction and whose effect depends on the timing of the DA signal itself. This bidirectional update rule can account for the results from pharmacology and optogenetics as well as the behavioral effects of reward rate on interval timing and the temporal selectivity of striatal neurons. Hence, by adopting a single RPE interpretation of DA, our results take a step toward unifying computational theories of reinforcement learning and interval timing. NEW & NOTEWORTHY How does dopamine (DA) influence interval timing? A large body of pharmacological evidence has suggested that DA accelerates timekeeping mechanisms. However, recent optogenetic work has shown exactly the opposite effect. In this article, we relate DA’s role in timekeeping to its most established role, as a critical component of reinforcement learning. This allows us to derive a neurobiologically plausible framework that reconciles a large body of DA’s temporal effects, including pharmacological, behavioral, electrophysiological, and optogenetic.

scholarly journals Striatal dynamics explain duration judgments

2015 ◽  
Author(s):  
Thiago S. Gouvêa ◽  
Tiago Monteiro ◽  
Asma Motiwala ◽  
Sofia Soares ◽  
Christian K. Machens ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Basal Ganglia ◽  
Interval Timing ◽  
Time Dependent ◽  
Striatal Neurons ◽  
Input Structure ◽  
State Changes ◽  
Psychophysical Task

The striatum is an input structure of the basal ganglia implicated in several time-dependent functions including reinforcement learning, decision making, and interval timing. To determine whether striatal ensembles drive subjects' judgments of duration, we manipulated and recorded from striatal neurons in rats performing a duration categorization psychophysical task. We found that the dynamics of striatal neurons predicted duration judgments, and that simultaneously recorded ensembles could judge duration as well as the animal. Furthermore, striatal neurons were necessary for duration judgments, as muscimol infusions produced a specific impairment in animals' duration sensitivity. Lastly, we show that time as encoded by striatal populations ran faster or slower when rats judged a duration as longer or shorter, respectively. These results demonstrate that the speed with which striatal population state changes supports the fundamental ability of animals to judge the passage of time.

scholarly journals Striatal dynamics explain duration judgments

eLife ◽  
2015 ◽  
Vol 4 ◽  
Author(s):  
Thiago S Gouvêa ◽  
Tiago Monteiro ◽  
Asma Motiwala ◽  
Sofia Soares ◽  
Christian Machens ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Basal Ganglia ◽  
Interval Timing ◽  
Time Dependent ◽  
Striatal Neurons ◽  
Input Structure ◽  
State Changes ◽  
Psychophysical Task

The striatum is an input structure of the basal ganglia implicated in several time-dependent functions including reinforcement learning, decision making, and interval timing. To determine whether striatal ensembles drive subjects' judgments of duration, we manipulated and recorded from striatal neurons in rats performing a duration categorization psychophysical task. We found that the dynamics of striatal neurons predicted duration judgments, and that simultaneously recorded ensembles could judge duration as well as the animal. Furthermore, striatal neurons were necessary for duration judgments, as muscimol infusions produced a specific impairment in animals' duration sensitivity. Lastly, we show that time as encoded by striatal populations ran faster or slower when rats judged a duration as longer or shorter, respectively. These results demonstrate that the speed with which striatal population state changes supports the fundamental ability of animals to judge the passage of time.

scholarly journals Dopamine mediates the bidirectional update of interval timing

2021 ◽  
Author(s):  
Anthony M.V. Jakob ◽  
John G Mikhael ◽  
Allison E Hamilos ◽  
John A Assad ◽  
Samuel J Gershman
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Interval Timing ◽  
Substantia Nigra Pars Compacta ◽  
Subjective Time ◽  
Reward Prediction Error ◽  
Learning Tasks ◽  
Reward Prediction ◽  
Reward Delivery ◽  
Speed Up

The role of dopamine as a reward prediction error signal in reinforcement learning tasks has been well-established over the past decades. Recent work has shown that the reward prediction error interpretation can also account for the effects of dopamine on interval timing by controlling the speed of subjective time. According to this theory, the timing of the dopamine signal relative to reward delivery dictates whether subjective time speeds up or slows down: Early DA signals speed up subjective time and late signals slow it down. To test this bidirectional prediction, we reanalyzed measurements of dopaminergic neurons in the substantia nigra pars compacta of mice performing a self-timed movement task. Using the slope of ramping dopamine activity as a read-out of subjective time speed, we found that trial-by-trial changes in the slope could be predicted from the timing of dopamine activity on the previous trial. This result provides a key piece of evidence supporting a unified computational theory of reinforcement learning and interval timing.

scholarly journals Confirmation Bias in the Course of Instructed Reinforcement Learning in Schizophrenia-Spectrum Disorders

2022 ◽  
Vol 12 (1) ◽  
pp. 90
Author(s):  
Dorota Frydecka ◽  
Patryk Piotrowski ◽  
Tomasz Bielawski ◽  
Edyta Pawlak ◽  
Ewa Kłosińska ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Large Body ◽  
Confirmation Bias ◽  
P Value ◽  
Schizophrenia Spectrum Disorders ◽  
Learning Tasks ◽  
Learning Rates ◽  
Schizophrenia Spectrum ◽  
Healthy Control ◽  
Spectrum Disorders

A large body of research attributes learning deficits in schizophrenia (SZ) to the systems involved in value representation (prefrontal cortex, PFC) and reinforcement learning (basal ganglia, BG) as well as to the compromised connectivity of these regions. In this study, we employed learning tasks hypothesized to probe the function and interaction of the PFC and BG in patients with SZ-spectrum disorders in comparison to healthy control (HC) subjects. In the Instructed Probabilistic Selection task (IPST), participants received false instruction about one of the stimuli used in the course of probabilistic learning which creates confirmation bias, whereby the instructed stimulus is overvalued in comparison to its real experienced value. The IPST was administered to 102 patients with SZ and 120 HC subjects. We have shown that SZ patients and HC subjects were equally influenced by false instruction in reinforcement learning (RL) probabilistic task (IPST) (p-value = 0.441); however, HC subjects had significantly higher learning rates associated with the process of overcoming cognitive bias in comparison to SZ patients (p-value = 0.018). The behavioral results of our study could be hypothesized to provide further evidence for impairments in the SZ-BG circuitry; however, this should be verified by neurofunctional imaging studies.

scholarly journals Selective network discovery via deep reinforcement learning on embedded spaces

2021 ◽  
Vol 6 (1) ◽  
Author(s):  
Peter Morales ◽  
Rajmonda Sulo Caceres ◽  
Tina Eliassi-Rad
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Sequential Decision ◽  
Network Discovery ◽  
Learning Tasks ◽  
Partially Observed ◽  
Decision Making Problem ◽  
Resource Collection ◽  
Improved Performance ◽  
Discovery Algorithms

AbstractComplex networks are often either too large for full exploration, partially accessible, or partially observed. Downstream learning tasks on these incomplete networks can produce low quality results. In addition, reducing the incompleteness of the network can be costly and nontrivial. As a result, network discovery algorithms optimized for specific downstream learning tasks given resource collection constraints are of great interest. In this paper, we formulate the task-specific network discovery problem as a sequential decision-making problem. Our downstream task is selective harvesting, the optimal collection of vertices with a particular attribute. We propose a framework, called network actor critic (NAC), which learns a policy and notion of future reward in an offline setting via a deep reinforcement learning algorithm. The NAC paradigm utilizes a task-specific network embedding to reduce the state space complexity. A detailed comparative analysis of popular network embeddings is presented with respect to their role in supporting offline planning. Furthermore, a quantitative study is presented on various synthetic and real benchmarks using NAC and several baselines. We show that offline models of reward and network discovery policies lead to significantly improved performance when compared to competitive online discovery algorithms. Finally, we outline learning regimes where planning is critical in addressing sparse and changing reward signals.

scholarly journals Solving Channel Allocation by Reinforcement Learning in Cognitive Enabled Vehicular Ad Hoc Networks

2021 ◽  
Author(s):  
Yunfan Su
Keyword(s):  
Dynamic Programming ◽  
Reinforcement Learning ◽  
Optimal Policy ◽  
Ad Hoc ◽  
Transition Probabilities ◽  
Channel Allocation ◽  
Dynamic Programming Method ◽  
Learning Method ◽  
Time Intervals ◽  
Model Free

Vehicular ad hoc network (VANET) is a promising technique that improves traffic safety and transportation efficiency and provides a comfortable driving experience. However, due to the rapid growth of applications that demand channel resources, efficient channel allocation schemes are required to utilize the performance of the vehicular networks. In this thesis, two Reinforcement learning (RL)-based channel allocation methods are proposed for a cognitive enabled VANET environment to maximize a long-term average system reward. First, we present a model-based dynamic programming method, which requires the calculations of the transition probabilities and time intervals between decision epochs. After obtaining the transition probabilities and time intervals, a relative value iteration (RVI) algorithm is used to find the asymptotically optimal policy. Then, we propose a model-free reinforcement learning method, in which we employ an agent to interact with the environment iteratively and learn from the feedback to approximate the optimal policy. Simulation results show that our reinforcement learning method can acquire a similar performance to that of the dynamic programming while both outperform the greedy method.

scholarly journals Solving Channel Allocation by Reinforcement Learning in Cognitive Enabled Vehicular Ad Hoc Networks

2021 ◽  
Author(s):  
Yunfan Su
Keyword(s):  
Dynamic Programming ◽  
Reinforcement Learning ◽  
Optimal Policy ◽  
Ad Hoc ◽  
Transition Probabilities ◽  
Channel Allocation ◽  
Dynamic Programming Method ◽  
Learning Method ◽  
Time Intervals ◽  
Model Free

Vehicular ad hoc network (VANET) is a promising technique that improves traffic safety and transportation efficiency and provides a comfortable driving experience. However, due to the rapid growth of applications that demand channel resources, efficient channel allocation schemes are required to utilize the performance of the vehicular networks. In this thesis, two Reinforcement learning (RL)-based channel allocation methods are proposed for a cognitive enabled VANET environment to maximize a long-term average system reward. First, we present a model-based dynamic programming method, which requires the calculations of the transition probabilities and time intervals between decision epochs. After obtaining the transition probabilities and time intervals, a relative value iteration (RVI) algorithm is used to find the asymptotically optimal policy. Then, we propose a model-free reinforcement learning method, in which we employ an agent to interact with the environment iteratively and learn from the feedback to approximate the optimal policy. Simulation results show that our reinforcement learning method can acquire a similar performance to that of the dynamic programming while both outperform the greedy method.

scholarly journals Survey of Deep-Learning Approaches for Remote Sensing Observation Enhancement

Sensors ◽  
2019 ◽  
Vol 19 (18) ◽  
pp. 3929 ◽  
Author(s):  
Grigorios Tsagkatakis ◽  
Anastasia Aidini ◽  
Konstantina Fotiadou ◽  
Michalis Giannopoulos ◽  
Anastasia Pentari ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Deep Learning ◽  
Large Body ◽  
Super Resolution ◽  
Imaging Systems ◽  
Learning Approaches ◽  
Language Understanding ◽  
Learning Tasks ◽  
Trade Offs ◽  
Sensing Platforms

Deep Learning, and Deep Neural Networks in particular, have established themselves as the new norm in signal and data processing, achieving state-of-the-art performance in image, audio, and natural language understanding. In remote sensing, a large body of research has been devoted to the application of deep learning for typical supervised learning tasks such as classification. Less yet equally important effort has also been allocated to addressing the challenges associated with the enhancement of low-quality observations from remote sensing platforms. Addressing such channels is of paramount importance, both in itself, since high-altitude imaging, environmental conditions, and imaging systems trade-offs lead to low-quality observation, as well as to facilitate subsequent analysis, such as classification and detection. In this paper, we provide a comprehensive review of deep-learning methods for the enhancement of remote sensing observations, focusing on critical tasks including single and multi-band super-resolution, denoising, restoration, pan-sharpening, and fusion, among others. In addition to the detailed analysis and comparison of recently presented approaches, different research avenues which could be explored in the future are also discussed.

Sign in / Sign up

Export Citation Format

Share Document