scholarly journals Reinforcement learning in discrete and continuous domains applied to ship trajectory generation

2012 ◽  
Vol 19 (Special) ◽  
pp. 31-36 ◽  
Author(s):  
Andrzej Rak ◽  
Witold Gierusz
Keyword(s):  
Reinforcement Learning ◽  
Least Squares ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Trajectory Generation ◽  
Decision Processes ◽  
The Other ◽  
Q Learning ◽  
Continuous Domains

ABSTRACT This paper presents the application of the reinforcement learning algorithms to the task of autonomous determination of the ship trajectory during the in-harbour and harbour approaching manoeuvres. Authors used Markov decision processes formalism to build up the background of algorithm presentation. Two versions of RL algorithms were tested in the simulations: discrete (Q-learning) and continuous form (Least-Squares Policy Iteration). The results show that in both cases ship trajectory can be found. However discrete Q-learning algorithm suffered from many limitations (mainly curse of dimensionality) and practically is not applicable to the examined task. On the other hand, LSPI gave promising results. To be fully operational, proposed solution should be extended by taking into account ship heading and velocity and coupling with advanced multi-variable controller.

scholarly journals Minibatch Recursive Least Squares Q-Learning

2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Chunyuan Zhang ◽  
Qi Song ◽  
Zeng Meng
Keyword(s):  
Reinforcement Learning ◽  
Least Squares ◽  
Linear Function ◽  
Function Approximation ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Optimization Technique ◽  
Recursive Least Squares ◽  
Q Learning ◽  
Linear Function Approximation

The deep Q-network (DQN) is one of the most successful reinforcement learning algorithms, but it has some drawbacks such as slow convergence and instability. In contrast, the traditional reinforcement learning algorithms with linear function approximation usually have faster convergence and better stability, although they easily suffer from the curse of dimensionality. In recent years, many improvements to DQN have been made, but they seldom make use of the advantage of traditional algorithms to improve DQN. In this paper, we propose a novel Q-learning algorithm with linear function approximation, called the minibatch recursive least squares Q-learning (MRLS-Q). Different from the traditional Q-learning algorithm with linear function approximation, the learning mechanism and model structure of MRLS-Q are more similar to those of DQNs with only one input layer and one linear output layer. It uses the experience replay and the minibatch training mode and uses the agent’s states rather than the agent’s state-action pairs as the inputs. As a result, it can be used alone for low-dimensional problems and can be seamlessly integrated into DQN as the last layer for high-dimensional problems as well. In addition, MRLS-Q uses our proposed average RLS optimization technique, so that it can achieve better convergence performance whether it is used alone or integrated with DQN. At the end of this paper, we demonstrate the effectiveness of MRLS-Q on the CartPole problem and four Atari games and investigate the influences of its hyperparameters experimentally.

Path Planning Collision Avoidance using Reinforcement Learning

2020 ◽  
Author(s):  
Josias G. Batista ◽  
Felipe J. S. Vasconcelos ◽  
Kaio M. Ramos ◽  
Darielson A. Souza ◽  
José L. N. Silva
Keyword(s):  
Reinforcement Learning ◽  
Path Planning ◽  
Production Process ◽  
Collision Avoidance ◽  
Production Systems ◽  
Learning Algorithm ◽  
Computational Cost ◽  
Trajectory Generation ◽  
Industrial Robots ◽  
Q Learning

Industrial robots have grown over the years making production systems more and more efficient, requiring the need for efficient trajectory generation algorithms that optimize and, if possible, generate collision-free trajectories without interrupting the production process. In this work is presented the use of Reinforcement Learning (RL), based on the Q-Learning algorithm, in the trajectory generation of a robotic manipulator and also a comparison of its use with and without constraints of the manipulator kinematics, in order to generate collisionfree trajectories. The results of the simulations are presented with respect to the efficiency of the algorithm and its use in trajectory generation, a comparison of the computational cost for the use of constraints is also presented.

A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms

1999 ◽  
Vol 11 (8) ◽  
pp. 2017-2060 ◽  
Author(s):  
Csaba Szepesvári ◽  
Michael L. Littman
Keyword(s):  
Reinforcement Learning ◽  
Value Function ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Sequential Decision ◽  
Q Learning ◽  
Markov Games ◽  
Optimal Behavior ◽  
Risk Sensitive ◽  
Optimal Value

Reinforcement learning is the problem of generating optimal behavior in a sequential decision-making environment given the opportunity of interacting with it. Many algorithms for solving reinforcement-learning problems work by computing improved estimates of the optimal value function. We extend prior analyses of reinforcement-learning algorithms and present a powerful new theorem that can provide a unified analysis of such value-function-based reinforcement-learning algorithms. The usefulness of the theorem lies in how it allows the convergence of a complex asynchronous reinforcement-learning algorithm to be proved by verifying that a simpler synchronous algorithm converges. We illustrate the application of the theorem by analyzing the convergence of Q-learning, model-based reinforcement learning, Q-learning with multistate updates, Q-learning for Markov games, and risk-sensitive reinforcement learning.

scholarly journals Iterative SARSA: The Modified SARSA Algorithm for Finding the Optimal Path

2020 ◽  
Vol 8 (6) ◽  
pp. 4333-4338
Keyword(s):  
Reinforcement Learning ◽  
Comparative Analysis ◽  
Mobile Robots ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Optimal Path ◽  
Autonomous Mobile Robots ◽  
Current Standard ◽  
Q Learning ◽  
Better Than

This paper presents a thorough comparative analysis of various reinforcement learning algorithms used by autonomous mobile robots for optimal path finding and, we propose a new algorithm called Iterative SARSA for the same. The main objective of the paper is to differentiate between the Q-learning and SARSA, and modify the latter. These algorithms use either the on-policy or off-policy methods of reinforcement learning. For the on-policy method, we have used the SARSA algorithm and for the off-policy method, the Q-learning algorithm has been used. These algorithms also have an impacting effect on finding the shortest path possible for the robot. Based on the results obtained, we have concluded how our algorithm is better than the current standard reinforcement learning algorithms

scholarly journals A line follower robot implementation using Lego's Mindstorms Kit and Q-Learning

2012 ◽  
Vol 22 ◽  
pp. 113-118 ◽  
Author(s):  
Víctor Ricardo Cruz-Álvarez ◽  
Enrique Hidalgo-Peña ◽  
Hector-Gabriel Acosta-Mesa
Keyword(s):  
Reinforcement Learning ◽  
Mobile Robots ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Programming Environment ◽  
Q Learning

A common problem working with mobile robots is that programming phase could be a long, expensive and heavy process for programmers. The reinforcement learning algorithms offer one of the most general frameworks in learning subjects. This work presents an approach using the Q-Learning algorithm on a Lego robot in order for it to learn by itself how to follow a blackline drawn down on a white surface, using Matlab [5] as programming environment.

scholarly journals Weighted Double Q-learning

2017 ◽  
Author(s):  
Zongzhang Zhang ◽  
Zhiyuan Pan ◽  
Mykel J. Kochenderfer
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Independent Sets ◽  
The Other ◽  
Q Learning ◽  
Stochastic Environments ◽  
Action Value ◽  
Reinforcement Learning Algorithm

Q-learning is a popular reinforcement learning algorithm, but it can perform poorly in stochastic environments due to overestimating action values. Overestimation is due to the use of a single estimator that uses the maximum action value as an approximation for the maximum expected action value. To avoid overestimation in Q-learning, the double Q-learning algorithm was recently proposed, which uses the double estimator method. It uses two estimators from independent sets of experiences, with one estimator determining the maximizing action and the other providing the estimate of its value. Double Q-learning sometimes underestimates the action values. This paper introduces a weighted double Q-learning algorithm, which is based on the construction of the weighted double estimator, with the goal of balancing between the overestimation in the single estimator and the underestimation in the double estimator. Empirically, the new algorithm is shown to perform well on several MDP problems.

scholarly journals Choice of cargo delivery option in multimodal connection based on reinforcement learning

2021 ◽  
Vol 2131 (3) ◽  
pp. 032103
Author(s):  
A P Badetskii ◽  
O A Medved
Keyword(s):  
Artificial Intelligence ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Digital Technologies ◽  
Qualitative Assessment ◽  
Q Learning ◽  
Cargo Delivery ◽  
Route Option

Abstract The article discusses the issues of choosing a route and an option of cargo flows in multimodal connection in modern conditions. Taking into account active development of artificial intelligence and digital technologies in all types of production activities, it is proposed to use reinforcement learning algorithms to solve the problem. An analysis of the existing algorithms has been carried out, on the basis of which it was found that when choosing a route option for cargo in a multimodal connection, it would be useful to have a qualitative assessment of terminal states. To obtain such an estimate, the Q-learning algorithm was applied in the article, which showed sufficient convergence and efficiency.

scholarly journals Application of Deep Q-learning and double Deep Q-learning algorithms to the task of control an inverted pendulum

2020 ◽  
pp. 7-25
Author(s):  
Alla Evseenko ◽  
◽  
Dmitrii Romannikov ◽  
Keyword(s):  
Artificial Intelligence ◽  
Machine Learning ◽  
Reinforcement Learning ◽  
Inverted Pendulum ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Training Time ◽  
Q Learning ◽  
Research Areas ◽  
Wide Range

Today, such a branch of science as «artificial intelligence» is booming in the world. Systems built on the basis of artificial intelligence methods have the ability to perform functions that are traditionally considered the prerogative of man. Artificial intelligence has a wide range of research areas. One such area is machine learning. This article discusses the algorithms of one of the approaches of machine learning – reinforcement learning (RL), according to which a lot of research and development has been carried out over the past seven years. Development and research on this approach is mainly carried out to solve problems in Atari 2600 games or in other similar ones. In this article, reinforcement training will be applied to one of the dynamic objects – an inverted pendulum. As a model of this object, we consider a model of an inverted pendulum on a cart taken from the Gym library, which contains many models that are used to test and analyze reinforcement learning algorithms. The article describes the implementation and study of two algorithms from this approach, Deep Q-learning and Double Deep Q-learning. As a result, training, testing and training time graphs for each algorithm are presented, on the basis of which it is concluded that it is desirable to use the Double Deep Q-learning algorithm, because the training time is approximately 2 minutes and provides the best control for the model of an inverted pendulum on a cart.

scholarly journals Aircraft Maintenance Check Scheduling Using Reinforcement Learning

Aerospace ◽  
2021 ◽  
Vol 8 (4) ◽  
pp. 113
Author(s):  
Pedro Andrade ◽  
Catarina Silva ◽  
Bernardete Ribeiro ◽  
Bruno F. Santos
Keyword(s):  
Reinforcement Learning ◽  
Time Horizon ◽  
Learning Algorithm ◽  
Initial Conditions ◽  
Q Learning ◽  
Scheduling Policy ◽  
Real Scenario ◽  
Maintenance Plan ◽  
Small Disturbances

This paper presents a Reinforcement Learning (RL) approach to optimize the long-term scheduling of maintenance for an aircraft fleet. The problem considers fleet status, maintenance capacity, and other maintenance constraints to schedule hangar checks for a specified time horizon. The checks are scheduled within an interval, and the goal is to, schedule them as close as possible to their due date. In doing so, the number of checks is reduced, and the fleet availability increases. A Deep Q-learning algorithm is used to optimize the scheduling policy. The model is validated in a real scenario using maintenance data from 45 aircraft. The maintenance plan that is generated with our approach is compared with a previous study, which presented a Dynamic Programming (DP) based approach and airline estimations for the same period. The results show a reduction in the number of checks scheduled, which indicates the potential of RL in solving this problem. The adaptability of RL is also tested by introducing small disturbances in the initial conditions. After training the model with these simulated scenarios, the results show the robustness of the RL approach and its ability to generate efficient maintenance plans in only a few seconds.

scholarly journals Improved Q-Learning Algorithm Based on Approximate State Matching in Agricultural Plant Protection Environment

Entropy ◽  
2021 ◽  
Vol 23 (6) ◽  
pp. 737
Author(s):  
Fengjie Sun ◽  
Xianchang Wang ◽  
Rui Zhang
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Optimal Policy ◽  
Feasible Solution ◽  
Learning Algorithm ◽  
Plant Protection ◽  
Agricultural Plant ◽  
Q Learning ◽  
Aerial Vehicle ◽  
Optimal Action

An Unmanned Aerial Vehicle (UAV) can greatly reduce manpower in the agricultural plant protection such as watering, sowing, and pesticide spraying. It is essential to develop a Decision-making Support System (DSS) for UAVs to help them choose the correct action in states according to the policy. In an unknown environment, the method of formulating rules for UAVs to help them choose actions is not applicable, and it is a feasible solution to obtain the optimal policy through reinforcement learning. However, experiments show that the existing reinforcement learning algorithms cannot get the optimal policy for a UAV in the agricultural plant protection environment. In this work we propose an improved Q-learning algorithm based on similar state matching, and we prove theoretically that there has a greater probability for UAV choosing the optimal action according to the policy learned by the algorithm we proposed than the classic Q-learning algorithm in the agricultural plant protection environment. This proposed algorithm is implemented and tested on datasets that are evenly distributed based on real UAV parameters and real farm information. The performance evaluation of the algorithm is discussed in detail. Experimental results show that the algorithm we proposed can efficiently learn the optimal policy for UAVs in the agricultural plant protection environment.

Sign in / Sign up

Export Citation Format

Share Document