ACE (Actor–Critic–Explorer) paradigm for reinforcement learning in basal ganglia: Highlighting the role of subthalamic and pallidal nuclei

Songbirds and humans share the ability to adaptively modify their vocalizations based on sensory feedback. Prior studies have focused primarily on the role that auditory feedback plays in shaping vocal output throughout life. In contrast, it is unclear whether and how non-auditory information drives vocal plasticity. Here, we first used a reinforcement learning paradigm to establish that non-auditory feedback can drive vocal learning in adult songbirds. We then assessed the role of a songbird basal ganglia-thalamocortical pathway critical to auditory vocal learning in this novel form of vocal plasticity. We found that both this circuit and its dopaminergic inputs are necessary for non-auditory vocal learning, demonstrating that this pathway is not specialized exclusively for auditory-driven vocal learning. The ability of this circuit to use both auditory and non-auditory information to guide vocal learning may reflect a general principle for the neural systems that support vocal plasticity across species.

Download Full-text

ACE (Actor-Critic-Explorer) Paradigm for Reinforcement Learning in Basal Ganglia: Highlighting the Role of the Indirect Pathway

Advances in Cognitive Science ◽

10.4135/9788132107910.n6 ◽

2014 ◽

pp. 71-90

Author(s):

Denny Joseph ◽

Garipelli Gangadhar ◽

V. Srinivasa Chakravarthy

Keyword(s):

Reinforcement Learning ◽

Basal Ganglia ◽

Indirect Pathway

Download Full-text

THE ROLE OF THE BASAL GANGLIA IN EXPLORATION IN A NEURAL MODEL BASED ON REINFORCEMENT LEARNING

International Journal of Neural Systems ◽

10.1142/s0129065706000548 ◽

2006 ◽

Vol 16 (02) ◽

pp. 111-124 ◽

Cited By ~ 35

Author(s):

D. SRIDHARAN ◽

P. S. PRASHANTH ◽

V. S. CHAKRAVARTHY

Keyword(s):

Reinforcement Learning ◽

Basal Ganglia ◽

State Space ◽

Chaotic Systems ◽

Neural Model ◽

Learning System ◽

Substantia Nigra Pars Compacta ◽

Water Pool ◽

Oscillatory Dynamics

We present a computational model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated water pool experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system's state space. We model the Subthalamic Nucleus-Globus Pallidus externa (STN-GPe) segment of the basal ganglia as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes such as chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore state-space, we suggest that the complex exploratory dynamics of STN-GPe system in conjunction with dopamine-based reward signaling from the Substantia Nigra pars compacta (SNc) present the two key ingredients of a reinforcement learning system.

Download Full-text

A Reinforcement-learning Account of Tourette Syndrome

European Psychiatry ◽

10.1016/j.eurpsy.2017.01.083 ◽

2017 ◽

Vol 41 (S1) ◽

pp. S10-S10

Author(s):

T. Maia

Keyword(s):

Reinforcement Learning ◽

Basal Ganglia ◽

Tourette Syndrome ◽

Behavioral Therapy ◽

Mathematical Explanation ◽

Reversal Training ◽

Computational Theory ◽

Habit Reversal Training ◽

Habit Learning

BackgroundTourette syndrome (TS) has long been thought to involve dopaminergic disturbances, given the effectiveness of antipsychotics in diminishing tics. Molecular-imaging studies have, by and large, confirmed that there are specific alterations in the dopaminergic system in TS. In parallel, multiple lines of evidence have implicated the motor cortico-basal ganglia-thalamo-cortical (CBGTC) loop in TS. Finally, several studies demonstrate that patients with TS exhibit exaggerated habit learning. This talk will present a computational theory of TS that ties together these multiple findings.MethodsThe computational theory builds on computational reinforcement-learning models, and more specifically on a recent model of the role of the direct and indirect basal-ganglia pathways in learning from positive and negative outcomes, respectively.ResultsA model defined by a small set of equations that characterize the role of dopamine in modulating learning and excitability in the direct and indirect pathways explains, in an integrated way: (1) the role of dopamine in the development of tics; (2) the relation between dopaminergic disturbances, involvement of the motor CBGTC loop, and excessive habit learning in TS; (3) the mechanism of action of antipsychotics in TS; and (4) the psychological and neural mechanisms of action of habit-reversal training, the main behavioral therapy for TS.ConclusionsA simple computational model, thoroughly grounded on computational theory and basic-science findings concerning dopamine and the basal ganglia, provides an integrated, rigorous mathematical explanation for a broad range of empirical findings in TS.Disclosure of interestThe author has not supplied his declaration of competing interest.

Download Full-text

Contributions of the basal ganglia to action sequence learning and performance

10.31234/osf.io/qp247 ◽

2019 ◽

Author(s):

Eric Garr

Keyword(s):

Reinforcement Learning ◽

Basal Ganglia ◽

Sequence Learning ◽

Computational Framework ◽

Action Sequence ◽

Action Sequences ◽

The Hierarchical Structure ◽

And Performance ◽

Action Sequencing

Animals engage in intricately woven and choreographed action sequences that are constructed from trial-and-error learning. The mechanisms by which the brain links together individual actions which are later recalled as fluid chains of behavior are not fully understood, but there is broad consensus that the basal ganglia play a crucial role in this process. This paper presents a comprehensive review of the role of the basal ganglia in action sequencing, with a focus on whether the computational framework of reinforcement learning can capture key behavioral features of sequencing and the neural mechanisms that underlie them. While a simple neurocomputational model of reinforcement learning can capture key features of action sequence learning, this model is not sufficient to capture goal-directed control of sequences or their hierarchical representation. The hierarchical structure of action sequences, in particular, poses a challenge for building better models of action sequencing, and it is in this regard that further investigations into basal ganglia information processing may be informative.

Download Full-text