THE ROLE OF THE BASAL GANGLIA IN EXPLORATION IN A NEURAL MODEL BASED ON REINFORCEMENT LEARNING

2006 ◽  
Vol 16 (02) ◽  
pp. 111-124 ◽  
Author(s):  
D. SRIDHARAN ◽  
P. S. PRASHANTH ◽  
V. S. CHAKRAVARTHY
Keyword(s):  
Reinforcement Learning ◽  
Basal Ganglia ◽  
State Space ◽  
Chaotic Systems ◽  
Neural Model ◽  
Learning System ◽  
Substantia Nigra Pars Compacta ◽  
Water Pool ◽  
Oscillatory Dynamics

We present a computational model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated water pool experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system's state space. We model the Subthalamic Nucleus-Globus Pallidus externa (STN-GPe) segment of the basal ganglia as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes such as chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore state-space, we suggest that the complex exploratory dynamics of STN-GPe system in conjunction with dopamine-based reward signaling from the Substantia Nigra pars compacta (SNc) present the two key ingredients of a reinforcement learning system.

scholarly journals Shared mechanisms of auditory and non-auditory vocal learning in the songbird brain

2021 ◽  
Author(s):  
James McGregor ◽  
Abigail Grassler ◽  
Paul I. Jaffe ◽  
Amanda Louise Jacob ◽  
Michael Brainard ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Basal Ganglia ◽  
General Principle ◽  
Auditory Feedback ◽  
Sensory Feedback ◽  
Vocal Learning ◽  
Neural Systems ◽  
Auditory Information ◽  
Vocal Plasticity

Songbirds and humans share the ability to adaptively modify their vocalizations based on sensory feedback. Prior studies have focused primarily on the role that auditory feedback plays in shaping vocal output throughout life. In contrast, it is unclear whether and how non-auditory information drives vocal plasticity. Here, we first used a reinforcement learning paradigm to establish that non-auditory feedback can drive vocal learning in adult songbirds. We then assessed the role of a songbird basal ganglia-thalamocortical pathway critical to auditory vocal learning in this novel form of vocal plasticity. We found that both this circuit and its dopaminergic inputs are necessary for non-auditory vocal learning, demonstrating that this pathway is not specialized exclusively for auditory-driven vocal learning. The ability of this circuit to use both auditory and non-auditory information to guide vocal learning may reflect a general principle for the neural systems that support vocal plasticity across species.

scholarly journals Modeling Basal Ganglia for Understanding Parkinsonian Reaching Movements

2011 ◽  
Vol 23 (2) ◽  
pp. 477-516 ◽  
Author(s):  
K. N. Magdoom ◽  
D. Subramanian ◽  
V. S. Chakravarthy ◽  
B. Ravindran ◽  
Shun-ichi Amari ◽  
...  
Keyword(s):  
Basal Ganglia ◽  
Motor Cortex ◽  
Reaching Movements ◽  
Substantia Nigra Pars Compacta ◽  
Temporal Difference ◽  
Indirect Pathway ◽  
Dynamical Disease ◽  
Exploratory Movements ◽  
Dopamine Signal

We present a computational model that highlights the role of basal ganglia (BG) in generating simple reaching movements. The model is cast within the reinforcement learning (RL) framework with correspondence between RL components and neuroanatomy as follows: dopamine signal of substantia nigra pars compacta as the temporal difference error, striatum as the substrate for the critic, and the motor cortex as the actor. A key feature of this neurobiological interpretation is our hypothesis that the indirect pathway is the explorer. Chaotic activity, originating from the indirect pathway part of the model, drives the wandering, exploratory movements of the arm. Thus, the direct pathway subserves exploitation, while the indirect pathway subserves exploration. The motor cortex becomes more and more independent of the corrective influence of BG as training progresses. Reaching trajectories show diminishing variability with training. Reaching movements associated with Parkinson's disease (PD) are simulated by reducing dopamine and degrading the complexity of indirect pathway dynamics by switching it from chaotic to periodic behavior. Under the simulated PD conditions, the arm exhibits PD motor symptoms like tremor, bradykinesia and undershooting. The model echoes the notion that PD is a dynamical disease.

A Reinforcement-learning Account of Tourette Syndrome

2017 ◽  
Vol 41 (S1) ◽  
pp. S10-S10
Author(s):  
T. Maia
Keyword(s):  
Reinforcement Learning ◽  
Basal Ganglia ◽  
Tourette Syndrome ◽  
Behavioral Therapy ◽  
Mathematical Explanation ◽  
Reversal Training ◽  
Computational Theory ◽  
Habit Reversal Training ◽  
Habit Learning

BackgroundTourette syndrome (TS) has long been thought to involve dopaminergic disturbances, given the effectiveness of antipsychotics in diminishing tics. Molecular-imaging studies have, by and large, confirmed that there are specific alterations in the dopaminergic system in TS. In parallel, multiple lines of evidence have implicated the motor cortico-basal ganglia-thalamo-cortical (CBGTC) loop in TS. Finally, several studies demonstrate that patients with TS exhibit exaggerated habit learning. This talk will present a computational theory of TS that ties together these multiple findings.MethodsThe computational theory builds on computational reinforcement-learning models, and more specifically on a recent model of the role of the direct and indirect basal-ganglia pathways in learning from positive and negative outcomes, respectively.ResultsA model defined by a small set of equations that characterize the role of dopamine in modulating learning and excitability in the direct and indirect pathways explains, in an integrated way: (1) the role of dopamine in the development of tics; (2) the relation between dopaminergic disturbances, involvement of the motor CBGTC loop, and excessive habit learning in TS; (3) the mechanism of action of antipsychotics in TS; and (4) the psychological and neural mechanisms of action of habit-reversal training, the main behavioral therapy for TS.ConclusionsA simple computational model, thoroughly grounded on computational theory and basic-science findings concerning dopamine and the basal ganglia, provides an integrated, rigorous mathematical explanation for a broad range of empirical findings in TS.Disclosure of interestThe author has not supplied his declaration of competing interest.

scholarly journals Contributions of the basal ganglia to action sequence learning and performance

2019 ◽  
Author(s):  
Eric Garr
Keyword(s):  
Reinforcement Learning ◽  
Basal Ganglia ◽  
Sequence Learning ◽  
Computational Framework ◽  
Action Sequence ◽  
Action Sequences ◽  
The Hierarchical Structure ◽  
And Performance ◽  
Action Sequencing

Animals engage in intricately woven and choreographed action sequences that are constructed from trial-and-error learning. The mechanisms by which the brain links together individual actions which are later recalled as fluid chains of behavior are not fully understood, but there is broad consensus that the basal ganglia play a crucial role in this process. This paper presents a comprehensive review of the role of the basal ganglia in action sequencing, with a focus on whether the computational framework of reinforcement learning can capture key behavioral features of sequencing and the neural mechanisms that underlie them. While a simple neurocomputational model of reinforcement learning can capture key features of action sequence learning, this model is not sufficient to capture goal-directed control of sequences or their hierarchical representation. The hierarchical structure of action sequences, in particular, poses a challenge for building better models of action sequencing, and it is in this regard that further investigations into basal ganglia information processing may be informative.

scholarly journals The Dopaminergic Control of Movement-Evolutionary Considerations

2021 ◽  
Vol 22 (20) ◽  
pp. 11284
Author(s):  
Juan Pérez-Fernández ◽  
Marta Barandela ◽  
Cecilia Jiménez-López
Keyword(s):  
Basal Ganglia ◽  
Dopaminergic Neurons ◽  
Dopaminergic System ◽  
Great Part ◽  
Substantia Nigra Pars Compacta ◽  
Motor Deficits ◽  
Motor Responses ◽  
Early Vertebrates ◽  
Characteristic Motor

Dopamine is likely the most studied modulatory neurotransmitter, in great part due to characteristic motor deficits in Parkinson’s disease that arise after the degeneration of the dopaminergic neurons in the substantia nigra pars compacta (SNc). The SNc, together with the ventral tegmental area (VTA), play a key role modulating motor responses through the basal ganglia. In contrast to the large amount of existing literature addressing the mammalian dopaminergic system, comparatively little is known in other vertebrate groups. However, in the last several years, numerous studies have been carried out in basal vertebrates, allowing a better understanding of the evolution of the dopaminergic system, especially the SNc/VTA. We provide an overview of existing research in basal vertebrates, mainly focusing on lampreys, belonging to the oldest group of extant vertebrates. The lamprey dopaminergic system and its role in modulating motor responses have been characterized in significant detail, both anatomically and functionally, providing the basis for understanding the evolution of the SNc/VTA in vertebrates. When considered alongside results from other early vertebrates, data in lampreys show that the key role of the SNc/VTA dopaminergic neurons modulating motor responses through the basal ganglia was already well developed early in vertebrate evolution.

Sign in / Sign up

Export Citation Format

Share Document