scholarly journals Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

Sensors ◽  
2019 ◽  
Vol 19 (7) ◽  
pp. 1576 ◽  
Author(s):  
Xiaomao Zhou ◽  
Tao Bai ◽  
Yanbin Gao ◽  
Yuntao Han
Keyword(s):  
Reinforcement Learning ◽  
Unsupervised Learning ◽  
Learning Algorithm ◽  
Spatial Scales ◽  
Learning Performance ◽  
Head Direction ◽  
Topological Map ◽  
Topological Maps ◽  
Hierarchical Reinforcement Learning ◽  
Continuous State

Extensive studies have shown that many animals’ capability of forming spatial representations for self-localization, path planning, and navigation relies on the functionalities of place and head-direction (HD) cells in the hippocampus. Although there are numerous hippocampal modeling approaches, only a few span the wide functionalities ranging from processing raw sensory signals to planning and action generation. This paper presents a vision-based navigation system that involves generating place and HD cells through learning from visual images, building topological maps based on learned cell representations and performing navigation using hierarchical reinforcement learning. First, place and HD cells are trained from sequences of visual stimuli in an unsupervised learning fashion. A modified Slow Feature Analysis (SFA) algorithm is proposed to learn different cell types in an intentional way by restricting their learning to separate phases of the spatial exploration. Then, to extract the encoded metric information from these unsupervised learning representations, a self-organized learning algorithm is adopted to learn over the emerged cell activities and to generate topological maps that reveal the topology of the environment and information about a robot’s head direction, respectively. This enables the robot to perform self-localization and orientation detection based on the generated maps. Finally, goal-directed navigation is performed using reinforcement learning in continuous state spaces which are represented by the population activities of place cells. In particular, considering that the topological map provides a natural hierarchical representation of the environment, hierarchical reinforcement learning (HRL) is used to exploit this hierarchy to accelerate learning. The HRL works on different spatial scales, where a high-level policy learns to select subgoals and a low-level policy learns over primitive actions to specialize on the selected subgoals. Experimental results demonstrate that our system is able to navigate a robot to the desired position effectively, and the HRL shows a much better learning performance than the standard RL in solving our navigation tasks.

scholarly journals A Reward Optimization Method Based on Action Subrewards in Hierarchical Reinforcement Learning

2014 ◽  
Vol 2014 ◽  
pp. 1-6
Author(s):  
Yuchen Fu ◽  
Quan Liu ◽  
Xionghong Ling ◽  
Zhiming Cui
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Optimization Method ◽  
Curse Of Dimensionality ◽  
Convergence Speed ◽  
Learning Method ◽  
Trial And Error ◽  
State Spaces ◽  
Reward Function ◽  
Hierarchical Reinforcement Learning

Reinforcement learning (RL) is one kind of interactive learning methods. Its main characteristics are “trial and error” and “related reward.” A hierarchical reinforcement learning method based on action subrewards is proposed to solve the problem of “curse of dimensionality,” which means that the states space will grow exponentially in the number of features and low convergence speed. The method can reduce state spaces greatly and choose actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. Apply it to the online learning in Tetris game, and the experiment result shows that the convergence speed of this algorithm can be enhanced evidently based on the new method which combines hierarchical reinforcement learning algorithm and action subrewards. The “curse of dimensionality” problem is also solved to a certain extent with hierarchical method. All the performance with different parameters is compared and analyzed as well.

scholarly journals Multi-Task Deep Reinforcement Learning for Continuous Action Control

2017 ◽  
Author(s):  
Zhaoyang Yang ◽  
Kathryn Merrick ◽  
Hussein Abbass ◽  
Lianwen Jin
Keyword(s):  
Reinforcement Learning ◽  
Network Architecture ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Action Control ◽  
Learning Performance ◽  
Sensor Data ◽  
Continuous Action ◽  
Single Task ◽  
Multiple Tasks

In this paper, we propose a deep reinforcement learning algorithm to learn multiple tasks concurrently. A new network architecture is proposed in the algorithm which reduces the number of parameters needed by more than 75% per task compared to typical single-task deep reinforcement learning algorithms. The proposed algorithm and network fuse images with sensor data and were tested with up to 12 movement-based control tasks on a simulated Pioneer 3AT robot equipped with a camera and range sensors. Results show that the proposed algorithm and network can learn skills that are as good as the skills learned by a comparable single-task learning algorithm. Results also show that learning performance is consistent even when the number of tasks and the number of constraints on the tasks increased.

scholarly journals Induction and Exploitation of Subgoal Automata for Reinforcement Learning

2021 ◽  
Vol 70 ◽  
pp. 1031-1116
Author(s):  
Daniel Furelos-Blanco ◽  
Mark Law ◽  
Anders Jonsson ◽  
Krysia Broda ◽  
Alessandra Russo
Keyword(s):  
Reinforcement Learning ◽  
Symmetry Breaking ◽  
Inductive Logic ◽  
Search Space ◽  
Learning Performance ◽  
Programming System ◽  
Continuous State ◽  
Automaton Learning ◽  
Minimum Number ◽  
High Level

In this paper we present ISA, an approach for learning and exploiting subgoals in episodic reinforcement learning (RL) tasks. ISA interleaves reinforcement learning with the induction of a subgoal automaton, an automaton whose edges are labeled by the task’s subgoals expressed as propositional logic formulas over a set of high-level events. A subgoal automaton also consists of two special states: a state indicating the successful completion of the task, and a state indicating that the task has finished without succeeding. A state-of-the-art inductive logic programming system is used to learn a subgoal automaton that covers the traces of high-level events observed by the RL agent. When the currently exploited automaton does not correctly recognize a trace, the automaton learner induces a new automaton that covers that trace. The interleaving process guarantees the induction of automata with the minimum number of states, and applies a symmetry breaking mechanism to shrink the search space whilst remaining complete. We evaluate ISA in several gridworld and continuous state space problems using different RL algorithms that leverage the automaton structures. We provide an in-depth empirical analysis of the automaton learning performance in terms of the traces, the symmetry breaking and specific restrictions imposed on the final learnable automaton. For each class of RL problem, we show that the learned automata can be successfully exploited to learn policies that reach the goal, achieving an average reward comparable to the case where automata are not learned but handcrafted and given beforehand.

Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions

2017 ◽  
Author(s):  
Aijun Bai ◽  
Stuart Russell
Keyword(s):  
Reinforcement Learning ◽  
Hierarchical Structure ◽  
Learning Algorithm ◽  
State Of The Art ◽  
State Machines ◽  
Learning To Learn ◽  
Hierarchical Reinforcement Learning ◽  
Abstract Machines ◽  
Finite State ◽  
Q Values

In the context of hierarchical reinforcement learning, the idea of hierarchies of abstract machines (HAMs) is to write a partial policy as a set of hierarchical finite state machines with unspecified choice states, and use reinforcement learning to learn an optimal completion of this partial policy. Given a HAM with potentially deep hierarchical structure, there often exist many internal transitions where a machine calls another machine with the environment state unchanged. In this paper, we propose a new hierarchical reinforcement learning algorithm that discovers such internal transitions automatically, and shortcircuits them recursively in computation of Q values. The resulting HAMQ-INT algorithm outperforms the state of the art significantly on the benchmark Taxi domain and a much more complex RoboCup Keepaway domain.

Sign in / Sign up

Export Citation Format

Share Document