scholarly journals Car-following method based on inverse reinforcement learning for autonomous vehicle decision-making

2018 ◽  
Vol 15 (6) ◽  
pp. 172988141881716 ◽  
Author(s):  
Hongbo Gao ◽  
Guanya Shi ◽  
Guotao Xie ◽  
Bo Cheng
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Optimization Problems ◽  
Learning Algorithm ◽  
Autonomous Vehicle ◽  
Sequential Decision ◽  
Inverse Reinforcement Learning ◽  
Car Following ◽  
Reward Function ◽  
Automatic Driving

There are still some problems need to be solved though there are a lot of achievements in the fields of automatic driving. One of those problems is the difficulty of designing a car-following decision-making system for complex traffic conditions. In recent years, reinforcement learning shows the potential in solving sequential decision optimization problems. In this article, we establish the reward function R of each driver data based on the inverse reinforcement learning algorithm, and r visualization is carried out, and then driving characteristics and following strategies are analyzed. At last, we show the efficiency of the proposed method by simulation in a highway environment.

scholarly journals Decision-making method for vehicle longitudinal automatic driving based on reinforcement Q-learning

2019 ◽  
Vol 16 (3) ◽  
pp. 172988141985318
Author(s):  
Zhenhai Gao ◽  
Tianjun Sun ◽  
Hongwei Xiao
Keyword(s):  
Decision Making ◽  
Process Model ◽  
Driving Simulator ◽  
Learning Algorithm ◽  
Autonomous Driving ◽  
Sequential Decision ◽  
Q Learning ◽  
Reward Function ◽  
Automatic Driving ◽  
Independent Decision

In the development of autonomous driving, decision-making has become one of the technical difficulties. Traditional rule-based decision-making methods lack adaptive capacity when dealing with unfamiliar and complex traffic conditions. However, reinforcement learning shows the potential to solve sequential decision problems. In this article, an independent decision-making method based on reinforcement Q-learning is proposed. First, a Markov decision process model is established by analysis of car-following. Then, the state set and action set are designed by the synthesized consideration of driving simulator experimental results and driving risk principles. Furthermore, the reinforcement Q-learning algorithm is developed mainly based on the reward function and update function. Finally, the feasibility is verified through random simulation tests, and the improvement is made by comparative analysis with a traditional method.

Safety-aware Adversarial Inverse Reinforcement Learning (S-AIRL) for Highway Autonomous Driving

2022 ◽  
pp. 1-14
Author(s):  
Fangjian Li ◽  
John R Wagner ◽  
Yue Wang
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Risky Behaviors ◽  
Autonomous Driving ◽  
Inverse Reinforcement Learning ◽  
Safety Issues ◽  
Reward Function ◽  
Sampling Process ◽  
Safety Awareness ◽  
Driving Scenario

Abstract Inverse reinforcement learning (IRL) has been successfully applied in many robotics and autonomous driving studies without the need for hand-tuning a reward function. However, it suffers from safety issues. Compared to the reinforcement learning (RL) algorithms, IRL is even more vulnerable to unsafe situations as it can only infer the importance of safety based on expert demonstrations. In this paper, we propose a safety-aware adversarial inverse reinforcement learning algorithm (S-AIRL). First, the control barrier function (CBF) is used to guide the training of a safety critic, which leverages the knowledge of system dynamics in the sampling process without training an additional guiding policy. The trained safety critic is then integrated into the discriminator to help discern the generated data and expert demonstrations from the standpoint of safety. Finally, to further improve the safety awareness, a regulator is introduced in the loss function of the discriminator training to prevent the recovered reward function from assigning high rewards to the risky behaviors. We tested our S-AIRL in the highway autonomous driving scenario. Comparing to the original AIRL algorithm, with the same level of imitation learning (IL) performance, the proposed S-AIRL can reduce the collision rate by 32.6%.

scholarly journals Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications

2019 ◽  
Vol 33 ◽  
pp. 7749-7758
Author(s):  
Daniel S. Brown ◽  
Scott Niekum
Keyword(s):  
Reinforcement Learning ◽  
Set Cover ◽  
Sequential Decision ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Set Cover Problem ◽  
Efficient Approximation Algorithm ◽  
Minimum Number ◽  
Teaching Problem ◽  
Novel Applications

Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decisionmaking task. We formalize the problem of finding maximally informative demonstrations for IRL as a machine teaching problem where the goal is to find the minimum number of demonstrations needed to specify the reward equivalence class of the demonstrator. We extend previous work on algorithmic teaching for sequential decision-making tasks by showing a reduction to the set cover problem which enables an efficient approximation algorithm for determining the set of maximallyinformative demonstrations. We apply our proposed machine teaching algorithm to two novel applications: providing a lower bound on the number of queries needed to learn a policy using active IRL and developing a novel IRL algorithm that can learn more efficiently from informative demonstrations than a standard IRL approach.

On the use of the policy gradient and Hessian in inverse reinforcement learning

2020 ◽  
Vol 14 (1) ◽  
pp. 117-150
Author(s):  
Alberto Maria Metelli ◽  
Matteo Pirotta ◽  
Marcello Restelli
Keyword(s):  
Reinforcement Learning ◽  
Sequential Decision ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Model Free ◽  
Learning Speed ◽  
Policy Gradient ◽  
Continuous Domains ◽  
Learning Policies ◽  
Finite Domains

Reinforcement Learning (RL) is an effective approach to solve sequential decision making problems when the environment is equipped with a reward function to evaluate the agent’s actions. However, there are several domains in which a reward function is not available and difficult to estimate. When samples of expert agents are available, Inverse Reinforcement Learning (IRL) allows recovering a reward function that explains the demonstrated behavior. Most of the classic IRL methods, in addition to expert’s demonstrations, require sampling the environment to evaluate each reward function, that, in turn, is built starting from a set of engineered features. This paper is about a novel model-free IRL approach that does not require to specify a function space where to search for the expert’s reward function. Leveraging on the fact that the policy gradient needs to be zero for an optimal policy, the algorithm generates an approximation space for the reward function, in which a reward is singled out employing a second-order criterion. After introducing our approach for finite domains, we extend it to continuous ones. The empirical results, on both finite and continuous domains, show that the reward function recovered by our algorithm allows learning policies that outperform those obtained with the true reward function, in terms of learning speed.

A Novel Adaptive Sampling Strategy for Deep Reinforcement Learning

2021 ◽  
Vol 20 (02) ◽  
pp. 2150011
Author(s):  
Xingxing Liang ◽  
Li Chen ◽  
Yanghe Feng ◽  
Zhong Liu ◽  
Yang Ma ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Adaptive Sampling ◽  
Learning Algorithm ◽  
Sampling Strategy ◽  
Sequential Decision ◽  
Fixed Temperature ◽  
Sample Distribution ◽  
Intelligent Decision Making ◽  
Experience Replay

Reinforcement learning, as an effective method to solve complex sequential decision-making problems, plays an important role in areas such as intelligent decision-making and behavioral cognition. It is well known that the sample experience replay mechanism contributes to the development of current deep reinforcement learning by reusing past samples to improve the efficiency of samples. However, the existing priority experience replay mechanism changes the sample distribution in the sample set due to the higher sampling frequency assigned to a specific transition, and it cannot be applied to actor-critic and other on-policy reinforcement learning algorithm. To address this, we propose an adaptive factor based on TD-error, which further increases sample utilization by giving more attention weight to samples of larger TD-error, and embeds it flexibly into the original Deep Q Network and Advantage Actor-Critic algorithm to improve their performance. Then we carried out the performance evaluation for the proposed architecture in the context of CartPole-V1 and 6 environments of Atari game experiments, respectively, and the obtained results either on the conditions of fixed temperature or annealing temperature, when compared to those produced by the vanilla DQN and original A2C, highlight the advantages in cumulative rewards and climb speed of the improved algorithms.

Driver-like decision-making method for vehicle longitudinal autonomous driving based on deep reinforcement learning

2021 ◽  
pp. 095440702110630
Author(s):  
Zhenhai Gao ◽  
Xiangtong Yan ◽  
Fei Gao ◽  
Lei He
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Autonomous Driving ◽  
Decision Strategies ◽  
Reward Function ◽  
Human Driver ◽  
Reward Functions ◽  
A Current ◽  
Better Than

Decision-making is one of the key parts of the research on vehicle longitudinal autonomous driving. Considering the behavior of human drivers when designing autonomous driving decision-making strategies is a current research hotspot. In longitudinal autonomous driving decision-making strategies, traditional rule-based decision-making strategies are difficult to apply to complex scenarios. Current decision-making methods that use reinforcement learning and deep reinforcement learning construct reward functions designed with safety, comfort, and economy. Compared with human drivers, the obtained decision strategies still have big gaps. Focusing on the above problems, this paper uses the driver’s behavior data to design the reward function of the deep reinforcement learning algorithm through BP neural network fitting, and uses the deep reinforcement learning DQN algorithm and the DDPG algorithm to establish two driver-like longitudinal autonomous driving decision-making models. The simulation experiment compares the decision-making effect of the two models with the driver curve. The results shows that the two algorithms can realize driver-like decision-making, and the consistency of the DDPG algorithm and human driver behavior is higher than that of the DQN algorithm, the effect of the DDPG algorithm is better than the DQN algorithm.

Research on decision-making of autonomous vehicle following based on reinforcement learning method

2019 ◽  
Vol 46 (3) ◽  
pp. 444-452 ◽  
Author(s):  
Hongbo Gao ◽  
Guanya Shi ◽  
Kelong Wang ◽  
Guotao Xie ◽  
Yuchao Liu
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Learning System ◽  
Explicit Calculation ◽  
Gradient Descent Method ◽  
Learning Method ◽  
Content Type ◽  
Car Following

Purpose Over the past decades, there has been significant research effort dedicated to the development of autonomous vehicles. The decision-making system, which is responsible for driving safety, is one of the most important technologies for autonomous vehicles. The purpose of this study is the use of an intensive learning method combined with car-following data by a driving simulator to obtain an explanatory learning following algorithm and establish an anthropomorphic car-following model. Design/methodology/approach This paper proposed car-following method based on reinforcement learning for autonomous vehicles decision-making. An approximator is used to approximate the value function by determining state space, action space and state transition relationship. A gradient descent method is used to solve the parameter. Findings The effect of car-following on certain driving styles is initially achieved through the simulation of step conditions. The effect of car-following initially proves that the reinforcement learning system is more adaptive to car following and that it has certain explanatory and stability based on the explicit calculation of R. Originality/value The simulation results show that the car-following method based on reinforcement learning for autonomous vehicle decision-making realizes reliable car-following decision-making and has the advantages of simple sample, small amount of data, simple algorithm and good robustness.

scholarly journals Decision-Making for the Autonomous Navigation of Maritime Autonomous Surface Ships Based on Scene Division and Deep Reinforcement Learning

Sensors ◽  
2019 ◽  
Vol 19 (18) ◽  
pp. 4055 ◽  
Author(s):  
Zhang ◽  
Wang ◽  
Liu ◽  
Chen
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Collision Avoidance ◽  
Autonomous Navigation ◽  
Learning Algorithm ◽  
Q Learning ◽  
Reward Function ◽  
International Regulations ◽  
Convergence Trend ◽  
Decision Making Model

This research focuses on the adaptive navigation of maritime autonomous surface ships (MASSs) in an uncertain environment. To achieve intelligent obstacle avoidance of MASSs in a port, an autonomous navigation decision-making model based on hierarchical deep reinforcement learning is proposed. The model is mainly composed of two layers: the scene division layer and an autonomous navigation decision-making layer. The scene division layer mainly quantifies the sub-scenarios according to the International Regulations for Preventing Collisions at Sea (COLREG). This research divides the navigational situation of a ship into entities and attributes based on the ontology model and Protégé language. In the decision-making layer, we designed a deep Q-learning algorithm utilizing the environmental model, ship motion space, reward function, and search strategy to learn the environmental state in a quantized sub-scenario to train the navigation strategy. Finally, two sets of verification experiments of the deep reinforcement learning (DRL) and improved DRL algorithms were designed with Rizhao port as a study case. Moreover, the experimental data were analyzed in terms of the convergence trend, iterative path, and collision avoidance effect. The results indicate that the improved DRL algorithm could effectively improve the navigation safety and collision avoidance.

scholarly journals A Study of Continuous Maximum Entropy Deep Inverse Reinforcement Learning

2019 ◽  
Vol 2019 ◽  
pp. 1-8
Author(s):  
Xi-liang Chen ◽  
Lei Cao ◽  
Zhi-xiong Xu ◽  
Jun Lai ◽  
Chen-xi Li
Keyword(s):  
Reinforcement Learning ◽  
State Space ◽  
Maximum Entropy ◽  
Learning Algorithm ◽  
Action Space ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Continuous State Space ◽  
Hot Start ◽  
Continuous State

The assumption of IRL is that demonstrations are optimally acting in an environment. In the past, most of the work on IRL needed to calculate optimal policies for different reward functions. However, this requirement is difficult to satisfy in large or continuous state space tasks. Let alone continuous action space. We propose a continuous maximum entropy deep inverse reinforcement learning algorithm for continuous state space and continues action space, which realizes the depth cognition of the environment model by the way of reconstructing the reward function based on the demonstrations, and a hot start mechanism based on demonstrations to make the training process faster and better. We compare this new approach to well-known IRL algorithms using Maximum Entropy IRL, DDPG, hot start DDPG, etc. Empirical results on classical control environments on OpenAI Gym: MountainCarContinues-v0 show that our approach is able to learn policies faster and better.

scholarly journals An Overview of Inverse Reinforcement Learning Techniques

2021 ◽  
Author(s):  
Syed Ihtesham Hussain Shah ◽  
Giuseppe De Pietro
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Decision Process ◽  
Autonomous Agents ◽  
Theoretical Background ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Learning Techniques ◽  
Markov Decision ◽  
Potential Use

In decision-making problems reward function plays an important role in finding the best policy. Reinforcement Learning (RL) provides a solution for decision-making problems under uncertainty in an Intelligent Environment (IE). However, it is difficult to specify the reward function for RL agents in large and complex problems. To counter these problems an extension of RL problem named Inverse Reinforcement Learning (IRL) is introduced, where reward function is learned from expert demonstrations. IRL is appealing for its potential use to build autonomous agents, capable of modeling others, deprived of compromising in performance of the task. This approach of learning by demonstrations relies on the framework of Markov Decision Process (MDP). This article elaborates original IRL algorithms along with their close variants to mitigate challenges. The purpose of this paper is to highlight an overview and theoretical background of IRL in the field of Machine Learning (ML) and Artificial Intelligence (AI). We presented a brief comparison between different variants of IRL in this article.

Sign in / Sign up

Export Citation Format

Share Document