In order to solve the problems of rapid path planning and effective obstacle avoidance for autonomous underwater vehicle (AUV) in 2D underwater environment, this paper proposes a path planning algorithm based on reinforcement learning mechanism and particle swarm optimization (RMPSO). A feedback mechanism of reinforcement learning is embedded into the particle swarm optimization (PSO) algorithm by using the proposed RMPSO to improve the convergence speed and adaptive ability of the PSO. Then, the RMPSO integrates the velocity synthesis method with the Bezier curve to eliminate the influence of ocean currents and save energy for AUV. Finally, the path is developed rapidly and obstacles are avoided effectively by using the RMPSO. Simulation and experiment results show the superiority of the proposed method compared with traditional methods.
This article introduces a model-based reinforcement learning (RL) approach for continuous state and action spaces. While most RL methods try to find closed-form policies, the approach taken here employs numerical on-line optimization of control action sequences. First, a general method for reformulating RL problems as optimization tasks is provided. Subsequently, Particle Swarm Optimization (PSO) is applied to search for optimal solutions. This Particle Swarm Optimization Policy (PSO-P) is effective for high dimensional state spaces and does not require a priori assumptions about adequate policy representations. Furthermore, by translating RL problems into optimization tasks, the rich collection of real-world inspired RL benchmarks is made available for benchmarking numerical optimization techniques. The effectiveness of PSO-P is demonstrated on the two standard benchmarks: mountain car and cart pole.
PurposeTo solve the path planning problem of the intelligent driving vehicular, this paper designs a hybrid path planning algorithm based on optimized reinforcement learning (RL) and improved particle swarm optimization (PSO).Design/methodology/approachFirst, the authors optimized the hyper-parameters of RL to make it converge quickly and learn more efficiently. Then the authors designed a pre-set operation for PSO to reduce the calculation of invalid particles. Finally, the authors proposed a correction variable that can be obtained from the cumulative reward of RL; this revises the fitness of the individual optimal particle and global optimal position of PSO to achieve an efficient path planning result. The authors also designed a selection parameter system to help to select the optimal path.FindingsSimulation analysis and experimental test results proved that the proposed algorithm has advantages in terms of practicability and efficiency. This research also foreshadows the research prospects of RL in path planning, which is also the authors’ next research direction.Originality/valueThe authors designed a pre-set operation to reduce the participation of invalid particles in the calculation in PSO. And then, the authors designed a method to optimize hyper-parameters to improve learning efficiency of RL. And then they used RL trained PSO to plan path. The authors also proposed an optimal path evaluation system. This research also foreshadows the research prospects of RL in path planning, which is also the authors’ next research direction.
This chapter introduces a model-based reinforcement learning (RL) approach for continuous state and action spaces. While most RL methods try to find closed-form policies, the approach taken here employs numerical online optimization of control action sequences following the strategy of nonlinear model predictive control. First, a general method for reformulating RL problems as optimization tasks is provided. Subsequently, particle swarm optimization (PSO) is applied to search for optimal solutions. This PSO policy (PSO-P) is effective for high dimensional state spaces and does not require a priori assumptions about adequate policy representations. Furthermore, by translating RL problems into optimization tasks, the rich collection of real-world-inspired RL benchmarks is made available for benchmarking numerical optimization techniques. The effectiveness of PSO-P is demonstrated on two standard benchmarks mountain car and cart-pole swing-up and a new industry-inspired benchmark, the so-called industrial benchmark.
Abstract
Particle swarm optimization (PSO) method is a well-known optimization algorithm, which shows good performance in solving different optimization problems. However, PSO usually suffers from slow convergence. In this paper, a reinforcement learning method is used to enhance PSO in convergence by replacing the uniformly distributed random number in the updating function by a random number generated from a well-selected normal distribution. The mean and variance of the normal distribution are estimated from the current state of each individual through a policy net. The historic behavior of the swarm group is learned to update the policy net and guide the selection of parameters of the normal distribution. The proposed algorithm is tested with numerical test functions and the results show that the convergence rate of PSO can be improved with the proposed Reinforcement Learning method (RL-PSO).
Particle Swarm Optimization with Reinforcement Learning for the Prediction of CpG Islands in the Human Genome