scholarly journals Reinforcement Learning-Based Collision Avoidance Guidance Algorithm for Fixed-Wing UAVs

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Yu Zhao ◽  
Jifeng Guo ◽  
Chengchao Bai ◽  
Hongxing Zheng
Keyword(s):  
Reinforcement Learning ◽  
Collision Avoidance ◽  
Short Term Memory ◽  
Learning Algorithm ◽  
Variable Number ◽  
Collision Probability ◽  
Learning Framework ◽  
Markov Game ◽  
The Neural Network ◽  
Guidance Algorithm

A deep reinforcement learning-based computational guidance method is presented, which is used to identify and resolve the problem of collision avoidance for a variable number of fixed-wing UAVs in limited airspace. The cooperative guidance process is first analyzed for multiple aircraft by formulating flight scenarios using multiagent Markov game theory and solving it by machine learning algorithm. Furthermore, a self-learning framework is established by using the actor-critic model, which is proposed to train collision avoidance decision-making neural networks. To achieve higher scalability, the neural network is customized to incorporate long short-term memory networks, and a coordination strategy is given. Additionally, a simulator suitable for multiagent high-density route scene is designed for validation, in which all UAVs run the proposed algorithm onboard. Simulated experiment results from several case studies show that the real-time guidance algorithm can reduce the collision probability of multiple UAVs in flight effectively even with a large number of aircraft.

Robot obstacle avoidance system using deep reinforcement learning

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Xiaojun Zhu ◽  
Yinghao Liang ◽  
Hanxu Sun ◽  
Xueqian Wang ◽  
Bin Ren
Keyword(s):  
Reinforcement Learning ◽  
Collision Avoidance ◽  
Obstacle Avoidance ◽  
Learning Algorithm ◽  
Optimal Path ◽  
Environmental Parameters ◽  
Working Environment ◽  
Content Type ◽  
Practical Applications ◽  
Human Operators

Purpose Most manufacturing plants choose the easy way of completely separating human operators from robots to prevent accidents, but as a result, it dramatically affects the overall quality and speed that is expected from human–robot collaboration. It is not an easy task to ensure human safety when he/she has entered a robot’s workspace, and the unstructured nature of those working environments makes it even harder. The purpose of this paper is to propose a real-time robot collision avoidance method to alleviate this problem. Design/methodology/approach In this paper, a model is trained to learn the direct control commands from the raw depth images through self-supervised reinforcement learning algorithm. To reduce the effect of sample inefficiency and safety during initial training, a virtual reality platform is used to simulate a natural working environment and generate obstacle avoidance data for training. To ensure a smooth transfer to a real robot, the automatic domain randomization technique is used to generate randomly distributed environmental parameters through the obstacle avoidance simulation of virtual robots in the virtual environment, contributing to better performance in the natural environment. Findings The method has been tested in both simulations with a real UR3 robot for several practical applications. The results of this paper indicate that the proposed approach can effectively make the robot safety-aware and learn how to divert its trajectory to avoid accidents with humans within the workspace. Research limitations/implications The method has been tested in both simulations with a real UR3 robot in several practical applications. The results indicate that the proposed approach can effectively make the robot be aware of safety and learn how to change its trajectory to avoid accidents with persons within the workspace. Originality/value This paper provides a novel collision avoidance framework that allows robots to work alongside human operators in unstructured and complex environments. The method uses end-to-end policy training to directly extract the optimal path from the visual inputs for the scene.

Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios

2020 ◽  
Vol 39 (7) ◽  
pp. 856-892 ◽  
Author(s):  
Tingxiang Fan ◽  
Pinxin Long ◽  
Wenxi Liu ◽  
Jia Pan
Keyword(s):  
Reinforcement Learning ◽  
Collision Avoidance ◽  
Autonomous Navigation ◽  
Large Scale ◽  
Learning Algorithm ◽  
Free Action ◽  
Parameter Tuning ◽  
Movement Velocity ◽  
Robot Systems ◽  
Multi Robot

Developing a safe and efficient collision-avoidance policy for multiple robots is challenging in the decentralized scenarios where each robot generates its paths with limited observation of other robots’ states and intentions. Prior distributed multi-robot collision-avoidance systems often require frequent inter-robot communication or agent-level features to plan a local collision-free action, which is not robust and computationally prohibitive. In addition, the performance of these methods is not comparable with their centralized counterparts in practice. In this article, we present a decentralized sensor-level collision-avoidance policy for multi-robot systems, which shows promising results in practical applications. In particular, our policy directly maps raw sensor measurements to an agent’s steering commands in terms of the movement velocity. As a first step toward reducing the performance gap between decentralized and centralized methods, we present a multi-scenario multi-stage training framework to learn an optimal policy. The policy is trained over a large number of robots in rich, complex environments simultaneously using a policy-gradient-based reinforcement-learning algorithm. The learning algorithm is also integrated into a hybrid control framework to further improve the policy’s robustness and effectiveness. We validate the learned sensor-level collision-3avoidance policy in a variety of simulated and real-world scenarios with thorough performance evaluations for large-scale multi-robot systems. The generalization of the learned policy is verified in a set of unseen scenarios including the navigation of a group of heterogeneous robots and a large-scale scenario with 100 robots. Although the policy is trained using simulation data only, we have successfully deployed it on physical robots with shapes and dynamics characteristics that are different from the simulated agents, in order to demonstrate the controller’s robustness against the simulation-to-real modeling error. Finally, we show that the collision-avoidance policy learned from multi-robot navigation tasks provides an excellent solution for safe and effective autonomous navigation for a single robot working in a dense real human crowd. Our learned policy enables a robot to make effective progress in a crowd without getting stuck. More importantly, the policy has been successfully deployed on different types of physical robot platforms without tedious parameter tuning. Videos are available at https://sites.google.com/view/hybridmrca .

Path Planning Collision Avoidance using Reinforcement Learning

2020 ◽  
Author(s):  
Josias G. Batista ◽  
Felipe J. S. Vasconcelos ◽  
Kaio M. Ramos ◽  
Darielson A. Souza ◽  
José L. N. Silva
Keyword(s):  
Reinforcement Learning ◽  
Path Planning ◽  
Production Process ◽  
Collision Avoidance ◽  
Production Systems ◽  
Learning Algorithm ◽  
Computational Cost ◽  
Trajectory Generation ◽  
Industrial Robots ◽  
Q Learning

Industrial robots have grown over the years making production systems more and more efficient, requiring the need for efficient trajectory generation algorithms that optimize and, if possible, generate collision-free trajectories without interrupting the production process. In this work is presented the use of Reinforcement Learning (RL), based on the Q-Learning algorithm, in the trajectory generation of a robotic manipulator and also a comparison of its use with and without constraints of the manipulator kinematics, in order to generate collisionfree trajectories. The results of the simulations are presented with respect to the efficiency of the algorithm and its use in trajectory generation, a comparison of the computational cost for the use of constraints is also presented.

scholarly journals Multi-agent cooperation Q-learning algorithm based on constrained Markov Game

2020 ◽  
Vol 17 (2) ◽  
pp. 647-664
Author(s):  
Yangyang Ge ◽  
Fei Zhu ◽  
Wei Huang ◽  
Peiyao Zhao ◽  
Quan Liu
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Multi Agent System ◽  
Agent System ◽  
Action Function ◽  
Q Learning ◽  
State Action ◽  
Markov Game ◽  
Safety Constraints ◽  
Multi Agent

Multi-Agent system has broad application in real world, whose security performance, however, is barely considered. Reinforcement learning is one of the most important methods to resolve Multi-Agent problems. At present, certain progress has been made in applying Multi-Agent reinforcement learning to robot system, man-machine match, and automatic, etc. However, in the above area, an agent may fall into unsafe states where the agent may find it difficult to bypass obstacles, to receive information from other agents and so on. Ensuring the safety of Multi-Agent system is of great importance in the above areas where an agent may fall into dangerous states that are irreversible, causing great damage. To solve the safety problem, in this paper we introduce a Multi-Agent Cooperation Q-Learning Algorithm based on Constrained Markov Game. In this method, safety constraints are added to the set of actions, and each agent, when interacting with the environment to search for optimal values, should be restricted by the safety rules, so as to obtain an optimal policy that satisfies the security requirements. Since traditional Multi-Agent reinforcement learning algorithm is no more suitable for the proposed model in this paper, a new solution is introduced for calculating the global optimum state-action function that satisfies the safety constraints. We take advantage of the Lagrange multiplier method to determine the optimal action that can be performed in the current state based on the premise of linearizing constraint functions, under conditions that the state-action function and the constraint function are both differentiable, which not only improves the efficiency and accuracy of the algorithm, but also guarantees to obtain the global optimal solution. The experiments verify the effectiveness of the algorithm.

scholarly journals Contracts for Difference: A Reinforcement Learning Approach

2020 ◽  
Vol 13 (4) ◽  
pp. 78
Author(s):  
Nico Zengeler ◽  
Uwe Handmann
Keyword(s):  
Reinforcement Learning ◽  
Short Term Memory ◽  
Learning Agents ◽  
Learning Framework ◽  
Learning Agent ◽  
Markov Decision ◽  
Economic Trends ◽  
Model Size ◽  
Contracts For Difference ◽  
Partially Observable

We present a deep reinforcement learning framework for an automatic trading of contracts for difference (CfD) on indices at a high frequency. Our contribution proves that reinforcement learning agents with recurrent long short-term memory (LSTM) networks can learn from recent market history and outperform the market. Usually, these approaches depend on a low latency. In a real-world example, we show that an increased model size may compensate for a higher latency. As the noisy nature of economic trends complicates predictions, especially in speculative assets, our approach does not predict courses but instead uses a reinforcement learning agent to learn an overall lucrative trading policy. Therefore, we simulate a virtual market environment, based on historical trading data. Our environment provides a partially observable Markov decision process (POMDP) to reinforcement learners and allows the training of various strategies.

scholarly journals A Deep Reinforcement Learning Framework for Rebalancing Dockless Bike Sharing Systems

2019 ◽  
Vol 33 ◽  
pp. 1393-1400 ◽  
Author(s):  
Ling Pan ◽  
Qingpeng Cai ◽  
Zhixuan Fang ◽  
Pingzhong Tang ◽  
Longbo Huang
Keyword(s):  
Reinforcement Learning ◽  
Spatial Information ◽  
Learning Algorithm ◽  
Service Level ◽  
Divide And Conquer ◽  
Gradient Algorithm ◽  
Learning Framework ◽  
Imbalance Problem ◽  
Look Ahead ◽  
Bike Sharing

Bike sharing provides an environment-friendly way for traveling and is booming all over the world. Yet, due to the high similarity of user travel patterns, the bike imbalance problem constantly occurs, especially for dockless bike sharing systems, causing significant impact on service quality and company revenue. Thus, it has become a critical task for bike sharing operators to resolve such imbalance efficiently. In this paper, we propose a novel deep reinforcement learning framework for incentivizing users to rebalance such systems. We model the problem as a Markov decision process and take both spatial and temporal features into consideration. We develop a novel deep reinforcement learning algorithm called Hierarchical Reinforcement Pricing (HRP), which builds upon the Deep Deterministic Policy Gradient algorithm. Different from existing methods that often ignore spatial information and rely heavily on accurate prediction, HRP captures both spatial and temporal dependencies using a divide-and-conquer structure with an embedded localized module. We conduct extensive experiments to evaluate HRP, based on a dataset from Mobike, a major Chinese dockless bike sharing company. Results show that HRP performs close to the 24-timeslot look-ahead optimization, and outperforms state-of-the-art methods in both service level and bike distribution. It also transfers well when applied to unseen areas.

Enhanced Policy Adaptation Through Directed Explorative Learning

2015 ◽  
Vol 12 (03) ◽  
pp. 1550028 ◽  
Author(s):  
Rok Vuga ◽  
Bojan Nemec ◽  
Aleš Ude
Keyword(s):  
Reinforcement Learning ◽  
Iterative Learning Control ◽  
Optimization Problems ◽  
Learning Algorithm ◽  
Learning Control ◽  
Iterative Learning ◽  
Learning Framework ◽  
Policy Adaptation ◽  
Explorative Learning ◽  
Integrated Policy

In this paper, we propose an integrated policy learning framework that fuses iterative learning control (ILC) and reinforcement learning. Integration is accomplished at the exploration level of the reinforcement learning algorithm. The proposed algorithm combines fast convergence properties of iterative learning control and robustness of reinforcement learning. This way, the advantages of both approaches are retained while overcoming their respective limitations. The proposed approach was verified in simulation and in real robot experiments on three challenging motion optimization problems.

Deep Deterministic Policy Gradient for Navigation of Mobile Robots

2021 ◽  
Vol 40 (1) ◽  
pp. 349-361
Author(s):  
Junior Costa de Jesus ◽  
Jair Augusto Bottega ◽  
Marco Antonio de Souza Leite Cuadros ◽  
Daniel Fernando Tello Gamarra
Keyword(s):  
Reinforcement Learning ◽  
Mobile Robot ◽  
Learning Algorithm ◽  
Mobile Robot Navigation ◽  
Simulated Environments ◽  
The Neural Network ◽  
Policy Gradient ◽  
Reward Functions ◽  
Continuous Actions ◽  
Reinforcement Learning Algorithm

This article describes the use of the Deep Deterministic Policy Gradient network, a deep reinforcement learning algorithm, for mobile robot navigation. The neural network structure has as inputs laser range findings, angular and linear velocities of the robot, and position and orientation of the mobile robot with respect to a goal position. The outputs of the network will be the angular and linear velocities used as control signals for the robot. The experiments demonstrated that deep reinforcement learning’s techniques that uses continuous actions, are efficient for decision-making in a mobile robot. Nevertheless, the design of the reward functions constitutes an important issue in the performance of deep reinforcement learning algorithms. In order to show the performance of the Deep Reinforcement Learning algorithm, we have applied successfully the proposed architecture in simulated environments and in experiments with a real robot.

scholarly journals Reinforcement Learning-Based Complete Area Coverage Path Planning for a Modified hTrihex Robot

Sensors ◽  
2021 ◽  
Vol 21 (4) ◽  
pp. 1067 ◽  
Author(s):  
Koppaka Ganesh Sai Apuroop ◽  
Anh Vu Le ◽  
Mohan Rajesh Elara ◽  
Bing J. Sheu
Keyword(s):  
Reinforcement Learning ◽  
Short Term Memory ◽  
Learning Algorithm ◽  
Area Coverage ◽  
Current Paper ◽  
Coverage Problem ◽  
Entire Area ◽  
Maximum Area ◽  
Coverage Path Planning ◽  
Cleaning Robot

One of the essential attributes of a cleaning robot is to achieve complete area coverage. Current commercial indoor cleaning robots have fixed morphology and are restricted to clean only specific areas in a house. The results of maximum area coverage are sub-optimal in this case. Tiling robots are innovative solutions for such a coverage problem. These new kinds of robots can be deployed in the cases of cleaning, painting, maintenance, and inspection, which require complete area coverage. Tiling robots’ objective is to cover the entire area by reconfiguring to different shapes as per the area requirements. In this context, it is vital to have a framework that enables the robot to maximize the area coverage while minimizing energy consumption. That means it is necessary for the robot to cover the maximum area with the least number of shape reconfigurations possible. The current paper proposes a complete area coverage planning module for the modified hTrihex, a honeycomb-shaped tiling robot, based on the deep reinforcement learning technique. This framework simultaneously generates the tiling shapes and the trajectory with minimum overall cost. In this regard, a convolutional neural network (CNN) with long short term memory (LSTM) layer was trained using the actor-critic experience replay (ACER) reinforcement learning algorithm. The simulation results obtained from the current implementation were compared against the results that were generated through traditional tiling theory models that included zigzag, spiral, and greedy search schemes. The model presented in the current paper was also compared against other methods where this problem was considered as a traveling salesman problem (TSP) solved through genetic algorithm (GA) and ant colony optimization (ACO) approaches. Our proposed scheme generates a path with a minimized cost at a lesser time.

Sign in / Sign up

Export Citation Format

Share Document