scholarly journals Stochastic Adaptive Forwarding Strategy Based on Deep Reinforcement Learning for Secure Mobile Video Communications in NDN

2021 ◽  
Vol 2021 ◽  
pp. 1-13
Author(s):  
Bowei Hao ◽  
Guoyong Wang ◽  
Mingchuan Zhang ◽  
Junlong Zhu ◽  
Ling Xing ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Rapid Development ◽  
Gradient Algorithm ◽  
Communication Security ◽  
Delivery Time ◽  
Mobile Video ◽  
Video Communications ◽  
Policy Gradient ◽  
Forwarding Strategy ◽  
Network Status

Named Data Networking (NDN) can effectively deal with the rapid development of mobile video services. For NDN, selecting a suitable forwarding interface according to the current network status can improve the efficiency of mobile video communication and can also avoid attacks to improve communication security. For this reason, we propose a stochastic adaptive forwarding strategy based on deep reinforcement learning (SAF-DRL) for secure mobile video communications in NDN. For each available forwarding interface, we introduce the twin delayed deep deterministic policy gradient algorithm to obtain a more robust forwarding strategy. Moreover, we conduct various numerical experiments to validate the performance of SAF-DRL. Compared with BR, RFA, SAF, and AFSndn forwarding strategies, the results show that SAF-DRL can reduce the delivery time and the average number of lost packets to improve the performance of NDN.

scholarly journals Reinforcement-Learning-Based Asynchronous Formation Control Scheme for Multiple Unmanned Surface Vehicles

2021 ◽  
Vol 11 (2) ◽  
pp. 546
Author(s):  
Jiajia Xie ◽  
Rui Zhou ◽  
Yuan Liu ◽  
Jun Luo ◽  
Shaorong Xie ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Formation Control ◽  
Rapid Development ◽  
Gradient Algorithm ◽  
Robot System ◽  
Physical Relationship ◽  
Unmanned Surface Vehicles ◽  
Main Challenge ◽  
Control Scheme ◽  
Multi Robot

The high performance and efficiency of multiple unmanned surface vehicles (multi-USV) promote the further civilian and military applications of coordinated USV. As the basis of multiple USVs’ cooperative work, considerable attention has been spent on developing the decentralized formation control of the USV swarm. Formation control of multiple USV belongs to the geometric problems of a multi-robot system. The main challenge is the way to generate and maintain the formation of a multi-robot system. The rapid development of reinforcement learning provides us with a new solution to deal with these problems. In this paper, we introduce a decentralized structure of the multi-USV system and employ reinforcement learning to deal with the formation control of a multi-USV system in a leader–follower topology. Therefore, we propose an asynchronous decentralized formation control scheme based on reinforcement learning for multiple USVs. First, a simplified USV model is established. Simultaneously, the formation shape model is built to provide formation parameters and to describe the physical relationship between USVs. Second, the advantage deep deterministic policy gradient algorithm (ADDPG) is proposed. Third, formation generation policies and formation maintenance policies based on the ADDPG are proposed to form and maintain the given geometry structure of the team of USVs during movement. Moreover, three new reward functions are designed and utilized to promote policy learning. Finally, various experiments are conducted to validate the performance of the proposed formation control scheme. Simulation results and contrast experiments demonstrate the efficiency and stability of the formation control scheme.

Preceding vehicle following algorithm with human driving characteristics

2021 ◽  
pp. 095440702098154
Author(s):  
Feng Pan ◽  
Hong Bao
Keyword(s):  
Reinforcement Learning ◽  
Weight Vector ◽  
Gradient Algorithm ◽  
Inner Product ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Human Driver ◽  
Policy Gradient ◽  
Preceding Vehicle ◽  
Action Spaces

This paper proposes a new approach of using reinforcement learning (RL) to train an agent to perform the task of vehicle following with human driving characteristics. We refer to the ideal of inverse reinforcement learning to design the reward function of the RL model. The factors that need to be weighed in vehicle following were vectorized into reward vectors, and the reward function was defined as the inner product of the reward vector and weights. Driving data of human drivers was collected and analyzed to obtain the true reward function. The RL model was trained with the deterministic policy gradient algorithm because the state and action spaces are continuous. We adjusted the weight vector of the reward function so that the value vector of the RL model could continuously approach that of a human driver. After dozens of rounds of training, we selected the policy with the nearest value vector to that of a human driver and tested it in the PanoSim simulation environment. The results showed the desired performance for the task of an agent following the preceding vehicle safely and smoothly.

scholarly journals Deterministic Value-Policy Gradients

2020 ◽  
Vol 34 (04) ◽  
pp. 3316-3323
Author(s):  
Qingpeng Cai ◽  
Ling Pan ◽  
Pingzhong Tang
Keyword(s):  
Reinforcement Learning ◽  
State Of The Art ◽  
Learning Algorithms ◽  
Infinite Horizon ◽  
Gradient Algorithm ◽  
Continuous Control ◽  
Model Bias ◽  
Model Free ◽  
Policy Gradient ◽  
Analytical Gradients

Reinforcement learning algorithms such as the deep deterministic policy gradient algorithm (DDPG) has been widely used in continuous control tasks. However, the model-free DDPG algorithm suffers from high sample complexity. In this paper we consider the deterministic value gradients to improve the sample efficiency of deep reinforcement learning algorithms. Previous works consider deterministic value gradients with the finite horizon, but it is too myopic compared with infinite horizon. We firstly give a theoretical guarantee of the existence of the value gradients in this infinite setting. Based on this theoretical guarantee, we propose a class of the deterministic value gradient algorithm (DVG) with infinite horizon, and different rollout steps of the analytical gradients by the learned model trade off between the variance of the value gradients and the model bias. Furthermore, to better combine the model-based deterministic value gradient estimators with the model-free deterministic policy gradient estimator, we propose the deterministic value-policy gradient (DVPG) algorithm. We finally conduct extensive experiments comparing DVPG with state-of-the-art methods on several standard continuous control benchmarks. Results demonstrate that DVPG substantially outperforms other baselines.

scholarly journals Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient

2019 ◽  
Vol 33 ◽  
pp. 4213-4220 ◽  
Author(s):  
Shihui Li ◽  
Yi Wu ◽  
Xinyue Cui ◽  
Honghua Dong ◽  
Fei Fang ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Gradient Algorithm ◽  
Training Environment ◽  
Local Optima ◽  
Continuous Action ◽  
Agent Learning ◽  
Policy Gradient ◽  
Multi Agent ◽  
Continuous Actions ◽  
Computational Intractability

Despite the recent advances of deep reinforcement learning (DRL), agents trained by DRL tend to be brittle and sensitive to the training environment, especially in the multi-agent scenarios. In the multi-agent setting, a DRL agent’s policy can easily get stuck in a poor local optima w.r.t. its training partners – the learned policy may be only locally optimal to other agents’ current policies. In this paper, we focus on the problem of training robust DRL agents with continuous actions in the multi-agent learning setting so that the trained agents can still generalize when its opponents’ policies alter. To tackle this problem, we proposed a new algorithm, MiniMax Multi-agent Deep Deterministic Policy Gradient (M3DDPG) with the following contributions: (1) we introduce a minimax extension of the popular multi-agent deep deterministic policy gradient algorithm (MADDPG), for robust policy learning; (2) since the continuous action space leads to computational intractability in our minimax learning objective, we propose Multi-Agent Adversarial Learning (MAAL) to efficiently solve our proposed formulation. We empirically evaluate our M3DDPG algorithm in four mixed cooperative and competitive multi-agent environments and the agents trained by our method significantly outperforms existing baselines.

scholarly journals A Method of Personalized Driving Decision for Smart Car Based on Deep Reinforcement Learning

Information ◽  
2020 ◽  
Vol 11 (6) ◽  
pp. 295 ◽  
Author(s):  
Xinpeng Wang ◽  
Chaozhong Wu ◽  
Jie Xue ◽  
Zhijun Chen
Keyword(s):  
Reinforcement Learning ◽  
Decision Model ◽  
Gradient Algorithm ◽  
Learning Goals ◽  
Learning Method ◽  
Automatic Driving ◽  
Proposed Model ◽  
Policy Gradient ◽  
Self Learning ◽  
Better Than

To date, automatic driving technology has become a hotspot in academia. It is necessary to provide a personalization of automatic driving decision for each passenger. The purpose of this paper is to propose a self-learning method for personalized driving decisions. First, collect and analyze driving data from different drivers to set learning goals. Then, Deep Deterministic Policy Gradient algorithm is utilized to design a driving decision system. Furthermore, personalized factors are introduced for some observed parameters to build a personalized driving decision model. Finally, compare the proposed method with classic Deep Reinforcement Learning algorithms. The results show that the performance of the personalized driving decision model is better than the classic algorithms, and it is similar to the manual driving situation. Therefore, the proposed model can effectively learn the human-like personalized driving decisions of different drivers for structured road. Based on this model, the smart car can accomplish personalized driving.

scholarly journals Efficient hindsight reinforcement learning using demonstrations for robotic tasks with sparse rewards

2020 ◽  
Vol 17 (1) ◽  
pp. 172988141989834
Author(s):  
Guoyu Zuo ◽  
Qishen Zhao ◽  
Jiahao Lu ◽  
Jiangeng Li
Keyword(s):  
Reinforcement Learning ◽  
Gradient Algorithm ◽  
Learning To Learn ◽  
Model Free ◽  
Learning Speed ◽  
Policy Gradient ◽  
Experience Replay ◽  
Speed Up ◽  
Reward Functions ◽  
Robotic Tasks

The goal of reinforcement learning is to enable an agent to learn by using rewards. However, some robotic tasks naturally specify with sparse rewards, and manually shaping reward functions is a difficult project. In this article, we propose a general and model-free approach for reinforcement learning to learn robotic tasks with sparse rewards. First, a variant of Hindsight Experience Replay, Curious and Aggressive Hindsight Experience Replay, is proposed to improve the sample efficiency of reinforcement learning methods and avoid the need for complicated reward engineering. Second, based on Twin Delayed Deep Deterministic policy gradient algorithm, demonstrations are leveraged to overcome the exploration problem and speed up the policy training process. Finally, the action loss is added into the loss function in order to minimize the vibration of output action while maximizing the value of the action. The experiments on simulated robotic tasks are performed with different hyperparameters to verify the effectiveness of our method. Results show that our method can effectively solve the sparse reward problem and obtain a high learning speed.

scholarly journals Deep Deterministic Policy Gradient Algorithm Based on Convolutional Block Attention for Autonomous Driving

Symmetry ◽  
2021 ◽  
Vol 13 (6) ◽  
pp. 1061
Author(s):  
Yanliang Jin ◽  
Qianhong Liu ◽  
Liquan Shen ◽  
Leiji Zhu
Keyword(s):  
Reinforcement Learning ◽  
Supervised Learning ◽  
State Of The Art ◽  
Autonomous Driving ◽  
Attention Mechanism ◽  
Gradient Algorithm ◽  
Excellent Performance ◽  
Average Speed ◽  
The Road ◽  
Policy Gradient

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

scholarly journals Revising the Observation Satellite Scheduling Problem Based on Deep Reinforcement Learning

2021 ◽  
Vol 13 (12) ◽  
pp. 2377
Author(s):  
Yixin Huang ◽  
Zhongcheng Mu ◽  
Shufan Wu ◽  
Benjie Cui ◽  
Yuxiao Duan
Keyword(s):  
Reinforcement Learning ◽  
Task Scheduling ◽  
Large Scale ◽  
Gradient Algorithm ◽  
Experimental Simulation ◽  
Scheduling Problem ◽  
Policy Gradient ◽  
Scheduling Method ◽  
Satellite Scheduling ◽  
Metaheuristic Optimization Algorithms

Earth observation satellite task scheduling research plays a key role in space-based remote sensing services. An effective task scheduling strategy can maximize the utilization of satellite resources and obtain larger objective observation profits. In this paper, inspired by the success of deep reinforcement learning in optimization domains, the deep deterministic policy gradient algorithm is adopted to solve a time-continuous satellite task scheduling problem. Moreover, an improved graph-based minimum clique partition algorithm is proposed for preprocessing in the task clustering phase by considering the maximum task priority and the minimum observation slewing angle under constraint conditions. Experimental simulation results demonstrate that the deep reinforcement learning-based task scheduling method is feasible and performs much better than traditional metaheuristic optimization algorithms, especially in large-scale problems.

scholarly journals Deep Ensemble Reinforcement Learning with Multiple Deep Deterministic Policy Gradient Algorithm

2020 ◽  
Vol 2020 ◽  
pp. 1-12 ◽  
Author(s):  
Junta Wu ◽  
Huiyun Li
Keyword(s):  
Reinforcement Learning ◽  
Autonomous Vehicles ◽  
Training Data ◽  
Gradient Algorithm ◽  
Distributed Data ◽  
Robot Arm ◽  
Training Time ◽  
Continuous Space ◽  
Policy Gradient ◽  
And Performance

Deep deterministic policy gradient algorithm operating over continuous space of actions has attracted great attention for reinforcement learning. However, the exploration strategy through dynamic programming within the Bayesian belief state space is rather inefficient even for simple systems. Another problem is the sequential and iterative training data with autonomous vehicles subject to the law of causality, which is against the i.i.d. (independent identically distributed) data assumption of the training samples. This usually results in failure of the standard bootstrap when learning an optimal policy. In this paper, we propose a framework of m-out-of-n bootstrapped and aggregated multiple deep deterministic policy gradient to accelerate the training process and increase the performance. Experiment results on the 2D robot arm game show that the reward gained by the aggregated policy is 10%–50% better than those gained by subpolicies. Experiment results on the open racing car simulator (TORCS) demonstrate that the new algorithm can learn successful control policies with less training time by 56.7%. Analysis on convergence is also given from the perspective of probability and statistics. These results verify that the proposed method outperforms the existing algorithms in both efficiency and performance.

scholarly journals Risk-Averse Trust Region Optimization for Reward-Volatility Reduction

2020 ◽  
Author(s):  
Lorenzo Bisi ◽  
Luca Sabbioni ◽  
Edoardo Vittori ◽  
Matteo Papini ◽  
Marcello Restelli
Keyword(s):  
Reinforcement Learning ◽  
Risk Measure ◽  
Trust Region ◽  
Gradient Algorithm ◽  
Expected Returns ◽  
Risk Averse ◽  
Return Variance ◽  
Policy Gradient ◽  
The Mean ◽  
Real Market

The use of reinforcement learning in algorithmic trading is of growing interest, since it offers the opportunity of making profit through the development of autonomous artificial traders, that do not depend on hard-coded rules. In such a framework, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in reinforcement learning through measures related to the distribution of returns. However, in trading it is essential to keep under control the risk of portfolio positions in the intermediate steps. In this paper, we define a novel measure of risk, which we call reward volatility, consisting of the variance of the rewards under the state-occupancy measure. This new risk measure is shown to bound the return variance so that reducing the former also constrains the latter. We derive a policy gradient theorem with a new objective function that exploits the mean-volatility relationship. Furthermore, we adapt TRPO, the well-known policy gradient algorithm with monotonic improvement guarantees, in a risk-averse manner. Finally, we test the proposed approach in two financial environments using real market data.

Sign in / Sign up

Export Citation Format

Share Document