Reinforcement Learning based Evolutionary Metric Filtering for High Dimensional Problems

Deep reinforcement learning (DRL) has gained great success by learning directly from high-dimensional sensory inputs, yet is notorious for the lack of interpretability. Interpretability of the subtasks is critical in hierarchical decision-making as it increases the transparency of black-box-style DRL approach and helps the RL practitioners to understand the high-level behavior of the system better. In this paper, we introduce symbolic planning into DRL and propose a framework of Symbolic Deep Reinforcement Learning (SDRL) that can handle both high-dimensional sensory inputs and symbolic planning. The task-level interpretability is enabled by relating symbolic actions to options. This framework features a planner – controller – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively. The three components cross-fertilize each other and eventually converge to an optimal symbolic plan along with the learned subtasks, bringing together the advantages of long-term planning capability with symbolic knowledge and end-to-end reinforcement learning directly from a high-dimensional sensory input. Experimental results validate the interpretability of subtasks, along with improved data efficiency compared with state-of-the-art approaches.

Download Full-text

Kernel dynamic policy programming: Practical reinforcement learning for high-dimensional robots

2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids) ◽

10.1109/humanoids.2016.7803345 ◽

2016 ◽

Cited By ~ 3

Author(s):

Yunduan Cui ◽

Takamitsu Matsubara ◽

Kenji Sugimoto

Keyword(s):

Reinforcement Learning ◽

High Dimensional ◽

Dynamic Policy

Download Full-text

Developing reinforcement learning for adaptive co-construction of continuous high-dimensional state and action spaces

Artificial Life and Robotics ◽

10.1007/s10015-012-0041-5 ◽

2012 ◽

Vol 17 (2) ◽

pp. 204-210 ◽

Cited By ~ 4

Author(s):

Masato Nagayoshi ◽

Hajime Murao ◽

Hisashi Tamaki

Keyword(s):

Reinforcement Learning ◽

High Dimensional ◽

Action Spaces

Download Full-text

Reward-Weighted Regression with Sample Reuse for Direct Policy Search in Reinforcement Learning

Neural Computation ◽

10.1162/neco_a_00199 ◽

2011 ◽

Vol 23 (11) ◽

pp. 2798-2832 ◽

Cited By ~ 11

Author(s):

Hirotaka Hachiya ◽

Jan Peters ◽

Masashi Sugiyama

Keyword(s):

Reinforcement Learning ◽

Expectation Maximization ◽

Search Method ◽

Weighted Regression ◽

High Dimensional ◽

Extended Version ◽

Conference Paper ◽

Policy Search ◽

Learning Framework ◽

Sampling Cost

Direct policy search is a promising reinforcement learning framework, in particular for controlling continuous, high-dimensional systems. Policy search often requires a large number of samples for obtaining a stable policy update estimator, and this is prohibitive when the sampling cost is expensive. In this letter, we extend an expectation-maximization-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, reward-weighted regression with sample reuse (R[Formula: see text]), is demonstrated through robot learning experiments. (This letter is an extended version of our earlier conference paper: Hachiya, Peters, & Sugiyama, 2009 .)

Download Full-text