scholarly journals Research on the Multiagent Joint Proximal Policy Optimization Algorithm Controlling Cooperative Fixed-Wing UAV Obstacle Avoidance

Sensors ◽  
2020 ◽  
Vol 20 (16) ◽  
pp. 4546
Author(s):  
Weiwei Zhao ◽  
Hairong Chu ◽  
Xikui Miao ◽  
Lihong Guo ◽  
Honghai Shen ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Attitude Control ◽  
Cooperative Control ◽  
Learning Algorithm ◽  
State Equations ◽  
Learning Agent ◽  
Environmental Adaptability ◽  
Decentralized Execution ◽  
Policy Optimization ◽  
Multi Uav

Multiple unmanned aerial vehicle (UAV) collaboration has great potential. To increase the intelligence and environmental adaptability of multi-UAV control, we study the application of deep reinforcement learning algorithms in the field of multi-UAV cooperative control. Aiming at the problem of a non-stationary environment caused by the change of learning agent strategy in reinforcement learning in a multi-agent environment, the paper presents an improved multiagent reinforcement learning algorithm—the multiagent joint proximal policy optimization (MAJPPO) algorithm with the centralized learning and decentralized execution. This algorithm uses the moving window averaging method to make each agent obtain a centralized state value function, so that the agents can achieve better collaboration. The improved algorithm enhances the collaboration and increases the sum of reward values obtained by the multiagent system. To evaluate the performance of the algorithm, we use the MAJPPO algorithm to complete the task of multi-UAV formation and the crossing of multiple-obstacle environments. To simplify the control complexity of the UAV, we use the six-degree of freedom and 12-state equations of the dynamics model of the UAV with an attitude control loop. The experimental results show that the MAJPPO algorithm has better performance and better environmental adaptability.

scholarly journals Machine Learning Combinatorial Frameworks for Architecture

2021 ◽  
Vol 7 (2) ◽  
pp. 20-29
Author(s):  
Joshua Lye ◽  
Alisa Andrasek
Keyword(s):  
Machine Learning ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Imitation Learning ◽  
Learning Models ◽  
Learning Agent ◽  
Policy Optimization ◽  
Machine Learning Models ◽  
Reinforcement Learning Algorithm

This paper investigates the application of machine learning for the simulation of larger architectural aggregations formed through the recombination of discrete components. This is primarily explored through establishing hardcoded assembly and connection logics which are used to form the framework of architectural fitness conditions for machine learning models. The key machine learning models researched are a combination of the deep reinforcement learning algorithm proximal policy optimization (PPO) and Generative Adversarial Imitation Learning (GAIL) in the Unity Machine Learning Agent asset toolkit. The goal of applying these machine learning models is to train the agent behaviours (discrete components) to learn specific logics of connection. In order to achieve assembled architectural `states that allow for spatial habitation through the process of simulation.

scholarly journals Hybrid Reinforcement Learning with Expert State Sequences

2019 ◽  
Vol 33 ◽  
pp. 3739-3746 ◽  
Author(s):  
Xiaoxiao Guo ◽  
Shiyu Chang ◽  
Mo Yu ◽  
Gerald Tesauro ◽  
Murray Campbell
Keyword(s):  
Neural Networks ◽  
Reinforcement Learning ◽  
Deep Neural Networks ◽  
Hybrid Approach ◽  
Imitation Learning ◽  
Learning Approaches ◽  
Inference Model ◽  
Learning Agent ◽  
Hybrid Reinforcement ◽  
Policy Optimization

Existing imitation learning approaches often require that the complete demonstration data, including sequences of actions and states, are available. In this paper, we consider a more realistic and difficult scenario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are unobserved. We propose a novel tensor-based model to infer the unobserved actions of the expert state sequences. The policy of the agent is then optimized via a hybrid objective combining reinforcement learning and imitation learning. We evaluated our hybrid approach on an illustrative domain and Atari games. The empirical results show that (1) the agents are able to leverage state expert sequences to learn faster than pure reinforcement learning baselines, (2) our tensor-based action inference model is advantageous compared to standard deep neural networks in inferring expert actions, and (3) the hybrid policy optimization objective is robust against noise in expert state sequences.

scholarly journals Accelerating the training of deep reinforcement learning in autonomous driving

2021 ◽  
Vol 10 (3) ◽  
pp. 649
Author(s):  
Emmanuel Ifeanyi Iroegbu ◽  
Devaraj Madhavi
Keyword(s):  
Reinforcement Learning ◽  
Autonomous Vehicle ◽  
Autonomous Driving ◽  
High Dimensional ◽  
Training Time ◽  
Learning Agent ◽  
Policy Gradient ◽  
Low Dimensional ◽  
Policy Optimization

Deep reinforcement learning has been successful in solving common autonomous driving tasks such as lane-keeping by simply using pixel data from the front view camera as input. However, raw pixel data contains a very high-dimensional observation that affects the learning quality of the agent due to the complexity imposed by a 'realistic' urban environment. Ergo, we investigate how compressing the raw pixel data from high-dimensional state to low-dimensional latent space offline using a variational autoencoder can significantly improve the training of a deep reinforcement learning agent. We evaluated our method on a simulated autonomous vehicle in car learning to act and compared our results with many baselines including deep deterministic policy gradient, proximal policy optimization, and soft actorcritic. The result shows that the method greatly accelerates the training time and there was a remarkable improvement in the quality of the deep reinforcement learning agent.

scholarly journals Reliability-Aware: Task Scheduling in Cloud Computing Using Multi-Agent Reinforcement Learning Algorithm and Neural Fitted Q

2020 ◽  
Vol 18 (1) ◽  
Keyword(s):  
Cloud Computing ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Buffer Size ◽  
Cloud Environment ◽  
Numerical Result ◽  
Learning Agent ◽  
Multi Agent ◽  
Reinforcement Learning Algorithm

Cloud computing becomes the basic alternative platform for the most users application in the recent years. The complexity increasing in cloud environment due to the continuous development of resources and applications needs a concentrated integrated fault tolerance approach to provide the quality of service. Focusing on reliability enhancement in an environment with dynamic changes such as cloud environment, we developed a multi-agent scheduler using Reinforcement Learning (RL) algorithm and Neural Fitted Q (NFQ) to effectively schedule the user requests. Our approach considers the queue buffer size for each resource by implementing the queue theory to design a queue model in a way that each scheduler agent has its own queue which receives the user requests from the global queue. A central learning agent responsible of learning the output of the scheduler agents and direct those scheduler agents through the feedback claimed from the previous step. The dynamicity problem in cloud environment is managed in our system by employing neural network which supports the reinforcement learning algorithm through a specified function. The numerical result demonstrated an efficiency of our proposed approach and enhanced the reliability

A formal methods approach to interpretable reinforcement learning for robotic planning

2019 ◽  
Vol 4 (37) ◽  
pp. eaay6276 ◽  
Author(s):  
Xiao Li ◽  
Zachary Serlin ◽  
Guang Yang ◽  
Calin Belta
Keyword(s):  
Reinforcement Learning ◽  
Formal Methods ◽  
Learning Algorithm ◽  
A Priori ◽  
Generation Process ◽  
Learning Approaches ◽  
Learning Agent ◽  
Domain Specific Knowledge ◽  
Robotic Planning ◽  
High Level

Growing interest in reinforcement learning approaches to robotic planning and control raises concerns of predictability and safety of robot behaviors realized solely through learned control policies. In addition, formally defining reward functions for complex tasks is challenging, and faulty rewards are prone to exploitation by the learning agent. Here, we propose a formal methods approach to reinforcement learning that (i) provides a formal specification language that integrates high-level, rich, task specifications with a priori, domain-specific knowledge; (ii) makes the reward generation process easily interpretable; (iii) guides the policy generation process according to the specification; and (iv) guarantees the satisfaction of the (critical) safety component of the specification. The main ingredients of our computational framework are a predicate temporal logic specifically tailored for robotic tasks and an automaton-guided, safe reinforcement learning algorithm based on control barrier functions. Although the proposed framework is quite general, we motivate it and illustrate it experimentally for a robotic cooking task, in which two manipulators worked together to make hot dogs.

Approximating Stackelberg Equilibrium in Anti-UAV Jamming Markov Game with Hierarchical Multi-Agent Deep Reinforcement Learning Algorithm

2021 ◽  
Author(s):  
Zikai Feng ◽  
Yuanyuan Wu ◽  
Mengxing Huang ◽  
Di Wu
Keyword(s):  
Reinforcement Learning ◽  
Stackelberg Game ◽  
Learning Algorithm ◽  
Computational Cost ◽  
Stackelberg Equilibrium ◽  
Large State Space ◽  
Comparable Performance ◽  
Aerial Vehicle ◽  
Multi Agent ◽  
Policy Optimization

Abstract In order to avoid the malicious jamming of the intelligent unmanned aerial vehicle (UAV) to ground users in the downlink communications, a new anti-UAV jamming strategy based on multi-agent deep reinforcement learning is studied in this paper. In this method, ground users aim to learn the best mobile strategies to avoid the jamming of UAV. The problem is modeled as a Stackelberg game to describe the competitive interaction between the UAV jammer (leader) and ground users (followers). To reduce the computational cost of equilibrium solution for the complex game with large state space, a hierarchical multi-agent proximal policy optimization (HMAPPO) algorithm is proposed to decouple the hybrid game into several sub-Markov games, which updates the actor and critic network of the UAV jammer and ground users at different time scales. Simulation results suggest that the hierarchical multi-agent proximal policy optimization -based anti-jamming strategy achieves comparable performance with lower time complexity than the benchmark strategies. The well-trained HMAPPO has the ability to obtain the optimal jamming strategy and the optimal anti-jamming strategies, which can approximate the Stackelberg equilibrium (SE).

scholarly journals Adaptive Supply Chain: Demand–Supply Synchronization Using Deep Reinforcement Learning

Algorithms ◽  
2021 ◽  
Vol 14 (8) ◽  
pp. 240
Author(s):  
Zhandos Kegenbekov ◽  
Ilya Jackson
Keyword(s):  
Supply Chain ◽  
Adaptive Control ◽  
Reinforcement Learning ◽  
Supply Chains ◽  
Optimization Algorithm ◽  
Base Stock ◽  
Base Stock Policy ◽  
Learning Agent ◽  
Policy Optimization ◽  
Stock Policy

Adaptive and highly synchronized supply chains can avoid a cascading rise-and-fall inventory dynamic and mitigate ripple effects caused by operational failures. This paper aims to demonstrate how a deep reinforcement learning agent based on the proximal policy optimization algorithm can synchronize inbound and outbound flows and support business continuity operating in the stochastic and nonstationary environment if end-to-end visibility is provided. The deep reinforcement learning agent is built upon the Proximal Policy Optimization algorithm, which does not require hardcoded action space and exhaustive hyperparameter tuning. These features, complimented with a straightforward supply chain environment, give rise to a general and task unspecific approach to adaptive control in multi-echelon supply chains. The proposed approach is compared with the base-stock policy, a well-known method in classic operations research and inventory control theory. The base-stock policy is prevalent in continuous-review inventory systems. The paper concludes with the statement that the proposed solution can perform adaptive control in complex supply chains. The paper also postulates fully fledged supply chain digital twins as a necessary infrastructural condition for scalable real-world applications.

scholarly journals Toward Self-Driving Bicycles Using State-of-the-Art Deep Reinforcement Learning Algorithms

Symmetry ◽  
2019 ◽  
Vol 11 (2) ◽  
pp. 290 ◽  
Author(s):  
SeungYoon Choi ◽  
Tuyen Le ◽  
Quang Nguyen ◽  
Md Layek ◽  
SeungGwan Lee ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Deep Neural Network ◽  
Learning Algorithm ◽  
State Of The Art ◽  
The Other ◽  
Gradient Algorithm ◽  
Reward Function ◽  
Policy Gradient ◽  
Policy Optimization ◽  
Start Location

In this paper, we propose a controller for a bicycle using the DDPG (Deep Deterministic Policy Gradient) algorithm, which is a state-of-the-art deep reinforcement learning algorithm. We use a reward function and a deep neural network to build the controller. By using the proposed controller, a bicycle can not only be stably balanced but also travel to any specified location. We confirm that the controller with DDPG shows better performance than the other baselines such as Normalized Advantage Function (NAF) and Proximal Policy Optimization (PPO). For the performance evaluation, we implemented the proposed algorithm in various settings such as fixed and random speed, start location, and destination location.

Queue Based Q-Learning for Efficient Resource Provisioning in Cloud Data Centers

2015 ◽  
Vol 11 (4) ◽  
pp. 37-54 ◽  
Author(s):  
A. Meera ◽  
S. Swamynathan
Keyword(s):  
Reinforcement Learning ◽  
Virtual Machine ◽  
Optimal Allocation ◽  
Virtual Machines ◽  
Learning Algorithm ◽  
Resource Provisioning ◽  
Time Step ◽  
Q Learning ◽  
Learning Agent ◽  
Efficient Resource

Cloud Computing is a novel paradigm that offers virtual resources on demand through internet. Due to rapid demand to cloud resources, it is difficult to estimate the user's demand. As a result, the complexity of resource provisioning increases, which leads to the requirement of an adaptive resource provisioning. In this paper, the authors address the problem of efficient resource provisioning through Queue based Q-learning algorithm using reinforcement learning agent. Reinforcement learning has been proved in various domains for automatic control and resource provisioning. In the absence of complete environment model, reinforcement learning can be used to define optimal allocation policies. The proposed Queue based Q-learning agent analyses the CPU utilization of all active Virtual Machines (VMs) and detects the least loaded virtual machine for resource provisioning. It detects the least loaded virtual machines through Inter Quartile Range. Using the queue size of virtual machines it looks ahead by one time step to find the optimal virtual machine for provisioning.

Sign in / Sign up

Export Citation Format

Share Document