Derivatives of Logarithmic Stationary Distributions for Policy Gradient Reinforcement Learning

2010 ◽  
Vol 22 (2) ◽  
pp. 342-376 ◽  
Author(s):  
Tetsuro Morimura ◽  
Eiji Uchibe ◽  
Junichiro Yoshimoto ◽  
Jan Peters ◽  
Kenji Doya
Keyword(s):  
Reinforcement Learning ◽  
Stationary State ◽  
Temporal Difference ◽  
Value Functions ◽  
Average Reward ◽  
Temporal Difference Learning ◽  
State Distribution ◽  
Learning Framework ◽  
Policy Gradient ◽  
Derivatives Of

Most conventional policy gradient reinforcement learning (PGRL) algorithms neglect (or do not explicitly make use of) a term in the average reward gradient with respect to the policy parameter. That term involves the derivative of the stationary state distribution that corresponds to the sensitivity of its distribution to changes in the policy parameter. Although the bias introduced by this omission can be reduced by setting the forgetting rate γ for the value functions close to 1, these algorithms do not permit γ to be set exactly at γ = 1. In this article, we propose a method for estimating the log stationary state distribution derivative (LSD) as a useful form of the derivative of the stationary state distribution through backward Markov chain formulation and a temporal difference learning framework. A new policy gradient (PG) framework with an LSD is also proposed, in which the average reward gradient can be estimated by setting γ = 0, so it becomes unnecessary to learn the value functions. We also test the performance of the proposed algorithms using simple benchmark tasks and show that these can improve the performances of existing PG methods.

On the Asymptotic Equivalence Between Differential Hebbian and Temporal Difference Learning

2009 ◽  
Vol 21 (4) ◽  
pp. 1173-1202 ◽  
Author(s):  
Christoph Kolodziejski ◽  
Bernd Porr ◽  
Florentin Wörgötter
Keyword(s):  
Reinforcement Learning ◽  
Hebbian Learning ◽  
Mathematical Proof ◽  
Temporal Difference ◽  
Temporal Difference Learning ◽  
Asymptotic Equivalence ◽  
Theoretical Contribution ◽  
Network Learning ◽  
Learning Framework

In this theoretical contribution, we provide mathematical proof that two of the most important classes of network learning—correlation-based differential Hebbian learning and reward-based temporal difference learning—are asymptotically equivalent when timing the learning with a modulatory signal. This opens the opportunity to consistently reformulate most of the abstract reinforcement learning framework from a correlation-based perspective more closely related to the biophysics of neurons.

scholarly journals An Insulin Bolus Advisor for Type 1 Diabetes Using Deep Reinforcement Learning

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5058 ◽  
Author(s):  
Taiyu Zhu ◽  
Kezhi Li ◽  
Lei Kuang ◽  
Pau Herrero ◽  
Pantelis Georgiou
Keyword(s):  
Type 1 Diabetes ◽  
Reinforcement Learning ◽  
Glucose Monitoring ◽  
Insulin Delivery ◽  
Target Range ◽  
Average Percentage ◽  
Learning Framework ◽  
Policy Gradient ◽  
Bolus Insulin

(1) Background: People living with type 1 diabetes (T1D) require self-management to maintain blood glucose (BG) levels in a therapeutic range through the delivery of exogenous insulin. However, due to the various variability, uncertainty and complex glucose dynamics, optimizing the doses of insulin delivery to minimize the risk of hyperglycemia and hypoglycemia is still an open problem. (2) Methods: In this work, we propose a novel insulin bolus advisor which uses deep reinforcement learning (DRL) and continuous glucose monitoring to optimize insulin dosing at mealtime. In particular, an actor-critic model based on deep deterministic policy gradient is designed to compute mealtime insulin doses. The proposed system architecture uses a two-step learning framework, in which a population model is first obtained and then personalized by subject-specific data. Prioritized memory replay is adopted to accelerate the training process in clinical practice. To validate the algorithm, we employ a customized version of the FDA-accepted UVA/Padova T1D simulator to perform in silico trials on 10 adult subjects and 10 adolescent subjects. (3) Results: Compared to a standard bolus calculator as the baseline, the DRL insulin bolus advisor significantly improved the average percentage time in target range (70–180 mg/dL) from 74.1%±8.4% to 80.9%±6.9% (p<0.01) and 54.9%±12.4% to 61.6%±14.1% (p<0.01) in the the adult and adolescent cohorts, respectively, while reducing hypoglycemia. (4) Conclusions: The proposed algorithm has the potential to improve mealtime bolus insulin delivery in people with T1D and is a feasible candidate for future clinical validation.

scholarly journals Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning

2019 ◽  
Author(s):  
Wenjie Shi ◽  
Shiji Song ◽  
Cheng Wu
Keyword(s):  
Reinforcement Learning ◽  
Maximum Entropy ◽  
Bellman Equation ◽  
Value Functions ◽  
Policy Actor ◽  
Model Free ◽  
Policy Gradient ◽  
Gradient Based ◽  
Continuous Actions ◽  
Stable Learning

Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we present an off-policy actor-critic, model-free maximum entropy deep RL algorithm called deep soft policy gradient (DSPG) by combining soft policy gradient with soft Bellman equation. To ensure stable learning while eliminating the need of two separate critics for soft value functions, we leverage double sampling approach to making the soft Bellman equation tractable. The experimental results demonstrate that our method outperforms in performance over off-policy prior methods.

scholarly journals Reward-Free Reinforcement Learning Algorithm Using Prediction Network

2020 ◽  
Author(s):  
Zhen Yu ◽  
Yimin Feng ◽  
Lijun Liu
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Value Functions ◽  
Learning Method ◽  
Reward Function ◽  
Network Training ◽  
Learning Tasks ◽  
Reward Value ◽  
Policy Gradient ◽  
Reward Functions

In general reinforcement learning tasks, the formulation of reward functions is a very important step in reinforcement learning. The reward function is not easy to formulate in a large number of systems. The network training effect is sensitive to the reward function, and different reward value functions will get different results. For a class of systems that meet specific conditions, the traditional reinforcement learning method is improved. A state quantity function is designed to replace the reward function, which is more efficient than the traditional reward function. At the same time, the predictive network link is designed so that the network can learn the value of the general state by using the special state. The overall structure of the network will be improved based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Finally, the algorithm was successfully applied in the environment of FrozenLake, and achieved good performance. The experiment proves the effectiveness of the algorithm and realizes rewardless reinforcement learning in a class of systems.

scholarly journals Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning with Polynomial Sample Complexity

2018 ◽  
Vol 63 ◽  
pp. 461-494
Author(s):  
Bo Liu ◽  
Ian Gemp ◽  
Mohammad Ghavamzadeh ◽  
Ji Liu ◽  
Sridhar Mahadevan ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Saddle Point ◽  
Temporal Difference ◽  
Temporal Difference Learning ◽  
Finite Sample ◽  
Approximation Techniques ◽  
Primal Dual ◽  
Improved Performance ◽  
Original Objective ◽  
Td Methods

In this paper, we introduce proximal gradient temporal difference learning, which provides a principled way of designing and analyzing true stochastic gradient temporal difference learning algorithms. We show how gradient TD (GTD) reinforcement learning methods can be formally derived, not by starting from their original objective functions, as previously attempted, but rather from a primal-dual saddle-point objective function. We also conduct a saddle-point error analysis to obtain finite-sample bounds on their performance. Previous analyses of this class of algorithms use stochastic approximation techniques to prove asymptotic convergence, and do not provide any finite-sample analysis. We also propose an accelerated algorithm, called GTD2-MP, that uses proximal "mirror maps" to yield an improved convergence rate. The results of our theoretical analysis imply that the GTD family of algorithms are comparable and may indeed be preferred over existing least squares TD methods for off-policy learning, due to their linear complexity. We provide experimental results showing the improved performance of our accelerated gradient TD methods.

Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning

2022 ◽  
pp. 1-20
Author(s):  
D. Xu ◽  
G. Chen
Keyword(s):  
Reinforcement Learning ◽  
Safety Factor ◽  
Cooperative Control ◽  
Learning Framework ◽  
Control Stage ◽  
Autonomous Planning ◽  
Operational Safety ◽  
Policy Gradient ◽  
Multi Agent ◽  
Reward Mechanism

Abstract In this paper, we expolore Multi-Agent Reinforcement Learning (MARL) methods for unmanned aerial vehicle (UAV) cluster. Considering that the current UAV cluster is still in the program control stage, the fully autonomous and intelligent cooperative combat has not been realised. In order to realise the autonomous planning of the UAV cluster according to the changing environment and cooperate with each other to complete the combat goal, we propose a new MARL framework. It adopts the policy of centralised training with decentralised execution, and uses Actor-Critic network to select the execution action and then to make the corresponding evaluation. The new algorithm makes three key improvements on the basis of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. The first is to improve learning framework; it makes the calculated Q value more accurate. The second is to add collision avoidance setting, which can increase the operational safety factor. And the third is to adjust reward mechanism; it can effectively improve the cluster’s cooperative ability. Then the improved MADDPG algorithm is tested by performing two conventional combat missions. The simulation results show that the learning efficiency is obviously improved, and the operational safety factor is further increased compared with the previous algorithm.

scholarly journals Hierarchical Average Reward Policy Gradient Algorithms (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13777-13778
Author(s):  
Akshay Dharmavaram ◽  
Matthew Riemer ◽  
Shalabh Bhatnagar
Keyword(s):  
General Purpose ◽  
Value Functions ◽  
Average Reward ◽  
Credit Assignment ◽  
Gradient Algorithms ◽  
Policy Gradient ◽  
Average Reward Criterion ◽  
Temporal Abstractions ◽  
Learning Options

Option-critic learning is a general-purpose reinforcement learning (RL) framework that aims to address the issue of long term credit assignment by leveraging temporal abstractions. However, when dealing with extended timescales, discounting future rewards can lead to incorrect credit assignments. In this work, we address this issue by extending the hierarchical option-critic policy gradient theorem for the average reward criterion. Our proposed framework aims to maximize the long-term reward obtained in the steady-state of the Markov chain defined by the agent's policy. Furthermore, we use an ordinary differential equation based approach for our convergence analysis and prove that the parameters of the intra-option policies, termination functions, and value functions, converge to their corresponding optimal values, with probability one. Finally, we illustrate the competitive advantage of learning options, in the average reward setting, on a grid-world environment with sparse rewards.

scholarly journals Choice-selective sequences dominate in cortical relative to thalamic inputs to nucleus accumbens, providing a potential substrate for credit assignment

2019 ◽  
Author(s):  
Nathan F. Parker ◽  
Avinash Baidya ◽  
Julia Cox ◽  
Laura Haetzel ◽  
Anna Zhukovskaya ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Nucleus Accumbens ◽  
Learning Task ◽  
Temporal Difference ◽  
Prelimbic Cortex ◽  
Temporal Difference Learning ◽  
Credit Assignment ◽  
Cortical Inputs ◽  
Selective Activity ◽  
Potential Substrate

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about what is represented in these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of temporal difference learning, a powerful algorithm to connect actions and outcomes across time. Finally, we test and confirm predictions of our circuit model by direct manipulation of nucleus accumbens input neurons. Thus, we integrate experiment and modeling to suggest a neural solution for credit assignment.

scholarly journals Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 3741-3748
Author(s):  
Kristopher De Asis ◽  
Alan Chan ◽  
Silviu Pitis ◽  
Richard Sutton ◽  
Daniel Graves
Keyword(s):  
Reinforcement Learning ◽  
Function Approximation ◽  
Value Function ◽  
Temporal Difference ◽  
Value Functions ◽  
Difference Methods ◽  
Td Methods ◽  
The Stability ◽  
The Value Function ◽  
Temporal Difference Methods

We explore fixed-horizon temporal difference (TD) methods, reinforcement learning algorithms for a new kind of value function that predicts the sum of rewards over a fixed number of future time steps. To learn the value function for horizon h, these algorithms bootstrap from the value function for horizon h−1, or some shorter horizon. Because no value function bootstraps from itself, fixed-horizon methods are immune to the stability problems that plague other off-policy TD methods using function approximation (also known as “the deadly triad”). Although fixed-horizon methods require the storage of additional value functions, this gives the agent additional predictive power, while the added complexity can be substantially reduced via parallel updates, shared weights, and n-step bootstrapping. We show how to use fixed-horizon value functions to solve reinforcement learning problems competitively with methods such as Q-learning that learn conventional value functions. We also prove convergence of fixed-horizon temporal difference methods with linear and general function approximation. Taken together, our results establish fixed-horizon TD methods as a viable new way of avoiding the stability problems of the deadly triad.

Sign in / Sign up

Export Citation Format

Share Document