Automatic Transfer Rate Estimation for Transfer Learning in Reinforcement Learning

2020 ◽  
Vol 2020 (0) ◽  
pp. 2A2-J03
Author(s):  
Hirokazu SATO ◽  
Ryoji OTSU ◽  
Yonghoon JI ◽  
Hiromitsu FUJII ◽  
Hitoshi KONO
Keyword(s):  
Reinforcement Learning ◽  
Transfer Learning ◽  
Transfer Rate ◽  
Rate Estimation

scholarly journals Automatic Transfer Rate Adjustment for Transfer Reinforcement Learning

2020 ◽  
Vol 11 (6) ◽  
pp. 47-54
Author(s):  
Hitoshi Kono ◽  
Yuto Sakamoto ◽  
Yonghoon Ji ◽  
Hiromitsu Fujii
Keyword(s):  
Reinforcement Learning ◽  
Transfer Learning ◽  
Computer Simulations ◽  
Transfer Rate ◽  
Sigmoid Function ◽  
Automatic Method ◽  
Target Task ◽  
Rate Adjustment ◽  
Robot Systems ◽  
Lock In

This paper proposes a novel parameter for transfer reinforcement learning to avoid over-fitting when an agent uses a transferred policy from a source task. Learning robot systems have recently been studied for many applications, such as home robots, communication robots, and warehouse robots. However, if the agent reuses the knowledge that has been sufficiently learned in the source task, deadlock may occur and appropriate transfer learning may not be realized. In the previous work, a parameter called transfer rate was proposed to adjust the ratio of transfer, and its contribution include avoiding dead lock in the target task. However, adjusting the parameter depends on human intuition and experiences. Furthermore, the method for deciding transfer rate has not discussed. Therefore, an automatic method for adjusting the transfer rate is proposed in this paper using a sigmoid function. Further, computer simulations are used to evaluate the effectiveness of the proposed method to improve the environmental adaptation performance in a target task, which refers to the situation of reusing knowledge.

scholarly journals Methods and Algorithms for Knowledge Reuse in Multiagent Reinforcement Learning

2020 ◽  
Author(s):  
Felipe Leno Da Silva ◽  
Anna Helena Reali Costa
Keyword(s):  
Reinforcement Learning ◽  
Transfer Learning ◽  
Learning Process ◽  
Trial And Error ◽  
Knowledge Reuse ◽  
Previous Knowledge ◽  
Learning Methods ◽  
Types Of Knowledge ◽  
Learning Agent ◽  
Multiagent Reinforcement Learning

Reinforcement Learning (RL) is a powerful tool that has been used to solve increasingly complex tasks. RL operates through repeated interactions of the learning agent with the environment, via trial and error. However, this learning process is extremely slow, requiring many interactions. In this thesis, we leverage previous knowledge so as to accelerate learning in multiagent RL problems. We propose knowledge reuse both from previous tasks and from other agents. Several flexible methods are introduced so that each of these two types of knowledge reuse is possible. This thesis adds important steps towards more flexible and broadly applicable multiagent transfer learning methods.

scholarly journals Applying a Deep Q Network for OpenAIs Car Racing Game

2020 ◽  
Author(s):  
Ali Fakhry
Keyword(s):  
Machine Learning ◽  
Reinforcement Learning ◽  
Transfer Learning ◽  
State Of The Art ◽  
Learning Techniques ◽  
Car Racing ◽  
Custom Made ◽  
Learning Technique ◽  
Reward Threshold

The applications of Deep Q-Networks are seen throughout the field of reinforcement learning, a large subsect of machine learning. Using a classic environment from OpenAI, CarRacing-v0, a 2D car racing environment, alongside a custom based modification of the environment, a DQN, Deep Q-Network, was created to solve both the classic and custom environments. The environments are tested using custom made CNN architectures and applying transfer learning from Resnet18. While DQNs were state of the art years ago, using it for CarRacing-v0 appears somewhat unappealing and not as effective as other reinforcement learning techniques. Overall, while the model did train and the agent learned various parts of the environment, attempting to reach the reward threshold for the environment with this reinforcement learning technique seems problematic and difficult as other techniques would be more useful.

Deep Reinforcement Learning for Transfer of Control Policies

2019 ◽  
Author(s):  
James D. Cunningham ◽  
Simon W. Miller ◽  
Michael A. Yukish ◽  
Timothy W. Simpson ◽  
Conrad S. Tucker
Keyword(s):  
Reinforcement Learning ◽  
Transfer Learning ◽  
Control Policy ◽  
Dynamic State ◽  
Test Case ◽  
Original Design ◽  
Control Policies ◽  
Parallel Learning ◽  
Rotor Design ◽  
Control Knowledge

Abstract We present a form-aware reinforcement learning (RL) method to extend control knowledge from one design form to another, without losing the ability to control the original design. A major challenge in developing control knowledge is the creation of generalized control policies across designs of varying form. Our presented RL policy is form-aware because in addition to receiving dynamic state information about the environment, it also receives states that encode information about the form of the design that is being controlled. In this paper, we investigate the impact of this mixed state space on transfer learning. We present a transfer learning method for extending a control policy to a different design form, while continuing to expose the agent to the original design during the training of the new design. To demonstrate this concept, we present a case study of a multi-rotor aircraft simulation, wherein the designated task is to achieve a stable hover. We show that by introducing form states, an RL agent is able to learn a control policy to achieve the hovering task with both a four rotor and three rotor design at once, whereas without the form states it can only hover with the four rotor design. We also benchmark our method against a test case that removes the transfer learning component, as well as a test case that removes the continued exposure to the original design to show the value of each of these components. We find that form states, transfer learning, and parallel learning all contribute to a more robust control policy for the new design, and that parallel learning is especially important for maintaining control knowledge of the original design.

scholarly journals An Introduction to Intertask Transfer for Reinforcement Learning

AI Magazine ◽  
2011 ◽  
Vol 32 (1) ◽  
pp. 15 ◽  
Author(s):  
Matthew E. Taylor ◽  
Peter Stone
Keyword(s):  
Reinforcement Learning ◽  
Transfer Learning ◽  
Learning Problem ◽  
Open Problems ◽  
Learning Framework ◽  
Learning Domains ◽  
Multiple Tasks ◽  
Exciting Area ◽  
Generalize Information ◽  
Selection Of

Transfer learning has recently gained popularity due to the development of algorithms that can successfully generalize information across multiple tasks. This article focuses on transfer in the context of reinforcement learning domains, a general learning framework where an agent acts in an environment to maximize a reward signal. The goals of this article are to (1) familiarize readers with the transfer learning problem in reinforcement learning domains, (2) explain why the problem is both interesting and difficult, (3) present a selection of existing techniques that demonstrate different solutions, and (4) provide representative open problems in the hope of encouraging additional research in this exciting area.

scholarly journals Combined Reinforcement Learning via Abstract Representations

2019 ◽  
Vol 33 ◽  
pp. 3582-3589 ◽  
Author(s):  
Vincent Francois-Lavet ◽  
Yoshua Bengio ◽  
Doina Precup ◽  
Joelle Pineau
Keyword(s):  
Reinforcement Learning ◽  
State Space ◽  
Transfer Learning ◽  
Computationally Efficient ◽  
Dimensional Representation ◽  
Learning Methods ◽  
Model Free ◽  
Abstract Representations ◽  
Low Dimensional ◽  
New Strategies

In the quest for efficient and robust reinforcement learning methods, both model-free and model-based approaches offer advantages. In this paper we propose a new way of explicitly bridging both approaches via a shared low-dimensional learned encoding of the environment, meant to capture summarizing abstractions. We show that the modularity brought by this approach leads to good generalization while being computationally efficient, with planning happening in a smaller latent state space. In addition, this approach recovers a sufficient low-dimensional representation of the environment, which opens up new strategies for interpretable AI, exploration and transfer learning.

Sign in / Sign up

Export Citation Format

Share Document