Value-Based Continuous Control Without Concrete State-Action Value Function

2021 ◽  
pp. 352-364
Author(s):  
Jin Zhu ◽  
Haixian Zhang ◽  
Zhen Pan
Keyword(s):  
Value Function ◽  
Continuous Control ◽  
State Action ◽  
Action Value

scholarly journals Reinforcement Learning for Optimizing Driving Policies on Cruising Taxis Services

2020 ◽  
Vol 12 (21) ◽  
pp. 8883
Author(s):  
Kun Jin ◽  
Wei Wang ◽  
Xuedong Hua ◽  
Wei Zhou
Keyword(s):  
Reinforcement Learning ◽  
Value Function ◽  
State Action ◽  
Future Reward ◽  
Long Run ◽  
Markov Decision ◽  
Action Value ◽  
Data Expansion ◽  
Taking Action ◽  
The Value Function

As the key element of urban transportation, taxis services significantly provide convenience and comfort for residents’ travel. However, the reality has not shown much efficiency. Previous researchers mainly aimed to optimize policies by order dispatch on ride-hailing services, which cannot be applied in cruising taxis services. This paper developed the reinforcement learning (RL) framework to optimize driving policies on cruising taxis services. Firstly, we formulated the drivers’ behaviours as the Markov decision process (MDP) progress, considering the influences after taking action in the long run. The RL framework using dynamic programming and data expansion was employed to calculate the state-action value function. Following the value function, drivers can determine the best choice and then quantify the expected future reward at a particular state. By utilizing historic orders data in Chengdu, we analysed the function value’s spatial distribution and demonstrated how the model could optimize the driving policies. Finally, the realistic simulation of the on-demand platform was built. Compared with other benchmark methods, the results verified that the new model performs better in increasing total revenue, answer rate and decreasing waiting time, with the relative percentages of 4.8%, 6.2% and −27.27% at most.

scholarly journals A solution for the Elevators Group Dispatch by Multiagent Reinforcement Learning

2019 ◽  
Author(s):  
Jordão Memória ◽  
José Maia
Keyword(s):  
Reinforcement Learning ◽  
Function Approximation ◽  
Value Function ◽  
The State ◽  
Evaluation Function ◽  
State Action ◽  
Traffic Pattern ◽  
Multiagent Reinforcement Learning ◽  
Multi Agent ◽  
Action Value

In this work, a modeling and algorithm based on multiagent reinforcement learning is developed for the problem of elevator group dispatch. The main advantage is that, along with the function approximation, this multi-agent solution leads to reduction of the state space, allowing complex states to be addressed with a synthesizing evaluation function. Each elevator is considered an agent that have to decide about two actions: answer or ignore the new call. With some iterations, the agents learn the weights of an evaluation function which approximate the state-action value function. The performance of solution (average waiting time - AWT), shown varying the traffic pattern, flow of people, number of elevators and number of floors, is comparable to other current proposals reported in the literature.

scholarly journals Energy Management of Hybrid UAV Based on Reinforcement Learning

Electronics ◽  
2021 ◽  
Vol 10 (16) ◽  
pp. 1929
Author(s):  
Huan Shen ◽  
Yao Zhang ◽  
Jianguo Mao ◽  
Zhiwei Yan ◽  
Linwei Wu
Keyword(s):  
Reinforcement Learning ◽  
State Space ◽  
Energy Management ◽  
Internal Combustion Engines ◽  
Value Function ◽  
Learning Algorithm ◽  
The State ◽  
Combustion Engines ◽  
State Action ◽  
Action Value

In order to solve the flight time problem of Unmanned Aerial Vehicles (UAV), this paper proposes a set of energy management strategies based on reinforcement learning for hybrid agricultural UAV. The battery is used to optimize the working point of internal combustion engines to the greatest extent while solving the high power demand issues of UAV and the response problem of internal combustion engines. Firstly, the decision-making oriented hybrid model and UAV dynamic model are established. Owing to the characteristics of the energy management strategy (EMS) based on reinforcement learning (RL), which is an intelligent optimization algorithm that has emerged in recent years, the complex theoretical formula derivation is avoided in the modeling process. In terms of the EMS, a double Q learning algorithm with strong convergence is adopted. The algorithm separates the state action value function database used in derivation decisions and the state action value function-updated database brought by the decision, so as to avoid delay and shock within the convergence process caused by maximum deviation. After the improvement, the off-line training is carried out with a large number of flight data generated in the past. The simulation results demonstrate that the improved algorithm can show better performance with less learning cost than before by virtue of the search function strategy proposed in this paper. In the state space, time-based and residual fuel-based selection are carried out successively, and the convergence rate and application effect are compared and analyzed. The results show that the learning algorithm has stronger robustness and convergence speed due to the appropriate selection of state space under different types of operating cycles. After 120,000 cycles of training, the fuel economy of the improved algorithm in this paper can reach more than 90% of that of the optimal solution, and can perform stably in actual flight.

Deep soccer analytics: learning an action-value function for evaluating soccer players

2020 ◽  
Vol 34 (5) ◽  
pp. 1531-1559
Author(s):  
Guiliang Liu ◽  
Yudong Luo ◽  
Oliver Schulte ◽  
Tarak Kharrat
Keyword(s):  
Value Function ◽  
Soccer Players ◽  
Action Value

Non-Markovian Reinforcement-Based on Self-Optimizing Memory Controller

2009 ◽  
Author(s):  
Hassab Elgawi Osman
Keyword(s):  
Value Function ◽  
Learning Capability ◽  
Memory Controller ◽  
State Action ◽  
Proposed Model ◽  
On Line ◽  
Memory Contents ◽  
Past Experiences ◽  
The Value Function

This paper contributes on designing robotic self-optimizing memory controller for non-Markovian reinforcement tasks. Rather than holistic search for the whole memory contents the model adopts associated feature analysis to successively memorize a newly event state-action pair as an action of past experience. Actor-Critic learning is used to adaptively tuning the control parameters, while on-line variant of random forests (RF) learner is used as memory-capable to approximate the policy of Actor and the value function of Critic. Learning capability of the proposed model is experimentally examined through non-markovian cart-pole balancing task. The result shows that our self-optimizing memory controller acquired complex behaviors such as balancing two poles simultaneously, displays long-term planning and generalization capacity based on past experiences.

scholarly journals Episodic Self-Imitation Learning with Hindsight

Electronics ◽  
2020 ◽  
Vol 9 (10) ◽  
pp. 1742
Author(s):  
Tianhong Dai ◽  
Hengyan Liu ◽  
Anil Anthony Bharath
Keyword(s):  
Learning Algorithm ◽  
Imitation Learning ◽  
Continuous Control ◽  
State Action ◽  
Good State ◽  
Agent Learning ◽  
Comparable Performance ◽  
Experience Replay ◽  
Speed Up ◽  
Action Spaces

Episodic self-imitation learning, a novel self-imitation algorithm with a trajectory selection module and an adaptive loss function, is proposed to speed up reinforcement learning. Compared to the original self-imitation learning algorithm, which samples good state–action pairs from the experience replay buffer, our agent leverages entire episodes with hindsight to aid self-imitation learning. A selection module is introduced to filter uninformative samples from each episode of the update. The proposed method overcomes the limitations of the standard self-imitation learning algorithm, a transitions-based method which performs poorly in handling continuous control environments with sparse rewards. From the experiments, episodic self-imitation learning is shown to perform better than baseline on-policy algorithms, achieving comparable performance to state-of-the-art off-policy algorithms in several simulated robot control tasks. The trajectory selection module is shown to prevent the agent learning undesirable hindsight experiences. With the capability of solving sparse reward problems in continuous control settings, episodic self-imitation learning has the potential to be applied to real-world problems that have continuous action spaces, such as robot guidance and manipulation.

scholarly journals Solving High-Dimensional Dynamic Portfolio Choice Models with Hierarchical B-Splines on Sparse Grids

2021 ◽  
Author(s):  
Peter Schober ◽  
Julian Valentin ◽  
Dirk Pflüger
Keyword(s):  
Portfolio Choice ◽  
Value Function ◽  
Choice Models ◽  
Sparse Grids ◽  
Continuous Control ◽  
B Splines ◽  
Gradient Based ◽  
Dynamic Portfolio ◽  
Dynamic Portfolio Choice ◽  
The Value Function

AbstractDiscrete time dynamic programming to solve dynamic portfolio choice models has three immanent issues: firstly, the curse of dimensionality prohibits more than a handful of continuous states. Secondly, in higher dimensions, even regular sparse grid discretizations need too many grid points for sufficiently accurate approximations of the value function. Thirdly, the models usually require continuous control variables, and hence gradient-based optimization with smooth approximations of the value function is necessary to obtain accurate solutions to the optimization problem. For the first time, we enable accurate and fast numerical solutions with gradient-based optimization while still allowing for spatial adaptivity using hierarchical B-splines on sparse grids. When compared to the standard linear bases on sparse grids or finite difference approximations of the gradient, our approach saves an order of magnitude in total computational complexity for a representative dynamic portfolio choice model with varying state space dimensionality, stochastic sample space, and choice variables.

Sign in / Sign up

Export Citation Format

Share Document