scholarly journals Algorithmic Improvements for Deep Reinforcement Learning Applied to Interactive Fiction

2020 ◽  
Vol 34 (04) ◽  
pp. 4328-4336
Author(s):  
Vishal Jain ◽  
William Fedus ◽  
Hugo Larochelle ◽  
Doina Precup ◽  
Marc G. Bellemare
Keyword(s):  
Reinforcement Learning ◽  
Structural Characteristics ◽  
Learning Algorithms ◽  
Learning Problem ◽  
Interactive Fiction ◽  
Reward Function ◽  
Learning Agent ◽  
Accumulated Reward ◽  
Partially Observable ◽  
Action Spaces

Text-based games are a natural challenge domain for deep reinforcement learning algorithms. Their state and action spaces are combinatorially large, their reward function is sparse, and they are partially observable: the agent is informed of the consequences of its actions through textual feedback. In this paper we emphasize this latter point and consider the design of a deep reinforcement learning agent that can play from feedback alone. Our design recognizes and takes advantage of the structural characteristics of text-based games. We first propose a contextualisation mechanism, based on accumulated reward, which simplifies the learning problem and mitigates partial observability. We then study different methods that rely on the notion that most actions are ineffectual in any given situation, following Zahavy et al.'s idea of an admissible action. We evaluate these techniques in a series of text-based games of increasing difficulty based on the TextWorld framework, as well as the iconic game Zork. Empirically, we find that these techniques improve the performance of a baseline deep reinforcement learning agent applied to text-based games.

Preceding vehicle following algorithm with human driving characteristics

2021 ◽  
pp. 095440702098154
Author(s):  
Feng Pan ◽  
Hong Bao
Keyword(s):  
Reinforcement Learning ◽  
Weight Vector ◽  
Gradient Algorithm ◽  
Inner Product ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Human Driver ◽  
Policy Gradient ◽  
Preceding Vehicle ◽  
Action Spaces

This paper proposes a new approach of using reinforcement learning (RL) to train an agent to perform the task of vehicle following with human driving characteristics. We refer to the ideal of inverse reinforcement learning to design the reward function of the RL model. The factors that need to be weighed in vehicle following were vectorized into reward vectors, and the reward function was defined as the inner product of the reward vector and weights. Driving data of human drivers was collected and analyzed to obtain the true reward function. The RL model was trained with the deterministic policy gradient algorithm because the state and action spaces are continuous. We adjusted the weight vector of the reward function so that the value vector of the RL model could continuously approach that of a human driver. After dozens of rounds of training, we selected the policy with the nearest value vector to that of a human driver and tested it in the PanoSim simulation environment. The results showed the desired performance for the task of an agent following the preceding vehicle safely and smoothly.

scholarly journals What Can You Do with a Rock? Affordance Extraction via Word Embeddings

2017 ◽  
Author(s):  
Nancy Fulda ◽  
Daniel Ricks ◽  
Ben Murdoch ◽  
David Wingate
Keyword(s):  
Reinforcement Learning ◽  
Computational Complexity ◽  
Linear Algebra ◽  
Autonomous Agents ◽  
Common Knowledge ◽  
Search Space ◽  
Word Embeddings ◽  
Knowledge Database ◽  
Learning Agent ◽  
Action Spaces

Autonomous agents must often detect affordances: the set of behaviors enabled by a situation. Affordance extraction is particularly helpful in domains with large action spaces, allowing the agent to prune its search space by avoiding futile behaviors. This paper presents a method for affordance extraction via word embeddings trained on a tagged Wikipedia corpus. The resulting word vectors are treated as a common knowledge database which can be queried using linear algebra. We apply this method to a reinforcement learning agent in a text-only environment and show that affordance-based action selection improves performance in most cases. Our method increases the computational complexity of each learning step but significantly reduces the total number of steps needed. In addition, the agent's action selections begin to resemble those a human would choose.

scholarly journals Universal Reinforcement Learning Algorithms: Survey and Experiments

2017 ◽  
Author(s):  
John Aslanides ◽  
Jan Leike ◽  
Marcus Hutter
Keyword(s):  
Reinforcement Learning ◽  
Open Source ◽  
Markov Decision Process ◽  
Decision Process ◽  
Empirical Investigation ◽  
State Of The Art ◽  
Learning Algorithms ◽  
Markov Decision ◽  
Reference Implementation ◽  
Partially Observable

Many state-of-the-art reinforcement learning (RL) algorithms typically assume that the environment is an ergodic Markov Decision Process (MDP). In contrast, the field of universal reinforcement learning (URL) is concerned with algorithms that make as few assumptions as possible about the environment. The universal Bayesian agent AIXI and a family of related URL algorithms have been developed in this setting. While numerous theoretical optimality results have been proven for these agents, there has been no empirical investigation of their behavior to date. We present a short and accessible survey of these URL algorithms under a unified notation and framework, along with results of some experiments that qualitatively illustrate some properties of the resulting policies, and their relative performance on partially-observable gridworld environments. We also present an open- source reference implementation of the algorithms which we hope will facilitate further understanding of, and experimentation with, these ideas.

scholarly journals A Monte-Carlo AIXI Approximation

2011 ◽  
Vol 40 ◽  
pp. 95-142 ◽  
Author(s):  
J. Veness ◽  
K.S. Ng ◽  
M. Hutter ◽  
W. Uther ◽  
D. Silver
Keyword(s):  
Monte Carlo ◽  
Reinforcement Learning ◽  
Search Algorithm ◽  
Future Research ◽  
Monte Carlo Tree Search ◽  
Practical Algorithms ◽  
Learning Agent ◽  
Context Tree ◽  
Direct Approximation ◽  
Partially Observable

This paper introduces a principled approach for the design of a scalable general reinforcement learning agent. Our approach is based on a direct approximation of AIXI, a Bayesian optimality notion for general reinforcement learning agents. Previously, it has been unclear whether the theory of AIXI could motivate the design of practical algorithms. We answer this hitherto open question in the affirmative, by providing the first computationally feasible approximation to the AIXI agent. To develop our approximation, we introduce a new Monte-Carlo Tree Search algorithm along with an agent-specific extension to the Context Tree Weighting algorithm. Empirically, we present a set of encouraging results on a variety of stochastic and partially observable domains. We conclude by proposing a number of directions for future research.

scholarly journals Contracts for Difference: A Reinforcement Learning Approach

2020 ◽  
Vol 13 (4) ◽  
pp. 78
Author(s):  
Nico Zengeler ◽  
Uwe Handmann
Keyword(s):  
Reinforcement Learning ◽  
Short Term Memory ◽  
Learning Agents ◽  
Learning Framework ◽  
Learning Agent ◽  
Markov Decision ◽  
Economic Trends ◽  
Model Size ◽  
Contracts For Difference ◽  
Partially Observable

We present a deep reinforcement learning framework for an automatic trading of contracts for difference (CfD) on indices at a high frequency. Our contribution proves that reinforcement learning agents with recurrent long short-term memory (LSTM) networks can learn from recent market history and outperform the market. Usually, these approaches depend on a low latency. In a real-world example, we show that an increased model size may compensate for a higher latency. As the noisy nature of economic trends complicates predictions, especially in speculative assets, our approach does not predict courses but instead uses a reinforcement learning agent to learn an overall lucrative trading policy. Therefore, we simulate a virtual market environment, based on historical trading data. Our environment provides a partially observable Markov decision process (POMDP) to reinforcement learners and allows the training of various strategies.

scholarly journals Object Affordance Driven Inverse Reinforcement Learning Through Conceptual Abstraction and Advice

2018 ◽  
Vol 9 (1) ◽  
pp. 277-294 ◽  
Author(s):  
Rupam Bhattacharyya ◽  
Shyamanta M. Hazarika
Keyword(s):  
Reinforcement Learning ◽  
High Dimensional ◽  
Inverse Reinforcement Learning ◽  
Intent Recognition ◽  
Reward Function ◽  
Object Affordances ◽  
Learning Agent ◽  
Markov Decision ◽  
Observed Behaviour ◽  
Object Affordance

Abstract Within human Intent Recognition (IR), a popular approach to learning from demonstration is Inverse Reinforcement Learning (IRL). IRL extracts an unknown reward function from samples of observed behaviour. Traditional IRL systems require large datasets to recover the underlying reward function. Object affordances have been used for IR. Existing literature on recognizing intents through object affordances fall short of utilizing its true potential. In this paper, we seek to develop an IRL system which drives human intent recognition along with the capability to handle high dimensional demonstrations exploiting the capability of object affordances. An architecture for recognizing human intent is presented which consists of an extended Maximum Likelihood Inverse Reinforcement Learning agent. Inclusion of Symbolic Conceptual Abstraction Engine (SCAE) along with an advisor allows the agent to work on Conceptually Abstracted Markov Decision Process. The agent recovers object affordance based reward function from high dimensional demonstrations. This function drives a Human Intent Recognizer through identification of probable intents. Performance of the resulting system on the standard CAD-120 dataset shows encouraging result.

scholarly journals CURIOSITY-DRIVEN REINFORCEMENT LEARNING AGENT FOR MAPPING UNKNOWN INDOOR ENVIRONMENTS

2021 ◽  
Vol V-1-2021 ◽  
pp. 129-136
Author(s):  
N. Botteghi ◽  
R. Schulte ◽  
B. Sirmacek ◽  
M. Poel ◽  
C. Brune
Keyword(s):  
Artificial Intelligence ◽  
Reinforcement Learning ◽  
Mobile Robot ◽  
Indoor Environments ◽  
Reward Function ◽  
Learning Framework ◽  
Learning Agent ◽  
The World ◽  
Reward Functions ◽  
Slam Algorithm

Abstract. Autonomously exploring and mapping is one of the open challenges of robotics and artificial intelligence. Especially when the environments are unknown, choosing the optimal navigation directive is not straightforward. In this paper, we propose a reinforcement learning framework for navigating, exploring, and mapping unknown environments. The reinforcement learning agent is in charge of selecting the commands for steering the mobile robot, while a SLAM algorithm estimates the robot pose and maps the environments. The agent, to select optimal actions, is trained to be curious about the world. This concept translates into the introduction of a curiosity-driven reward function that encourages the agent to steer the mobile robot towards unknown and unseen areas of the world and the map. We test our approach in explorations challenges in different indoor environments. The agent trained with the proposed reward function outperforms the agents trained with reward functions commonly used in the literature for solving such tasks.

scholarly journals Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 6672-6679 ◽  
Author(s):  
Deheng Ye ◽  
Zhao Liu ◽  
Mingfei Sun ◽  
Bei Shi ◽  
Peilin Zhao ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Large Scale ◽  
Action Control ◽  
Learning Problem ◽  
Complex Action ◽  
Complex Control ◽  
Learning Framework ◽  
High Scalability ◽  
Action Spaces ◽  
Level Performance

We study the reinforcement learning problem of complex action control in the Multi-player Online Battle Arena (MOBA) 1v1 games. This problem involves far more complicated state and action spaces than those of traditional 1v1 games, such as Go and Atari series, which makes it very difficult to search any policies with human-level performance. In this paper, we present a deep reinforcement learning framework to tackle this problem from the perspectives of both system and algorithm. Our system is of low coupling and high scalability, which enables efficient explorations at large scale. Our algorithm includes several novel strategies, including control dependency decoupling, action mask, target attention, and dual-clip PPO, with which our proposed actor-critic network can be effectively trained in our system. Tested on the MOBA game Honor of Kings, the trained AI agents can defeat top professional human players in full 1v1 games.

scholarly journals Safe Deployment of a Reinforcement Learning Robot Using Self Stabilization

2021 ◽  
Author(s):  
Nanda Kishore Sreenivas ◽  
Shrisha Rao
Keyword(s):  
Reinforcement Learning ◽  
Finite Number ◽  
Autonomous Vehicles ◽  
Safe Space ◽  
Training Phase ◽  
Prior Work ◽  
Industrial Systems ◽  
Learning Agent ◽  
Improved Performance ◽  
Action Spaces

In toy environments like video games, a reinforcement learning agent is deployed and operates within the same state space in which it was trained. However, in robotics applications such as industrial systems or autonomous vehicles, this cannot be guaranteed. A robot can be pushed out of its training space by some unforeseen perturbation, which may cause it to go into an unknown state from which it has not been trained to move towards its goal. While most prior work in the area of RL safety focuses on ensuring safety in the training phase, this paper focuses on ensuring the safe deployment of a robot that has already been trained to operate within a safe space. This work defines a condition on the state and action spaces, that if satisfied, guarantees the robot's recovery to safety independently. We also propose a strategy and design that facilitate this recovery within a finite number of steps after perturbation. This is implemented and tested against a standard RL model, and the results indicate a much-improved performance.

scholarly journals Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

2020 ◽  
Vol 10 (12) ◽  
pp. 4088
Author(s):  
Andreas Verleysen ◽  
Thomas Holvoet ◽  
Remko Proesmans ◽  
Cedric Den Haese ◽  
Francis wyffels
Keyword(s):  
Reinforcement Learning ◽  
Low Cost ◽  
Tactile Sensor ◽  
The State ◽  
Learning Needs ◽  
Deformable Objects ◽  
Deformable Object ◽  
Reward Function ◽  
Rectangular Patch ◽  
Learning Agent

Deformable objects such as ropes, wires, and clothing are omnipresent in society and industry but are little researched in robotics research. This is due to the infinite amount of possible state configurations caused by the deformations of the deformable object. Engineered approaches try to cope with this by implementing highly complex operations in order to estimate the state of the deformable object. This complexity can be circumvented by utilizing learning-based approaches, such as reinforcement learning, which can deal with the intrinsic high-dimensional state space of deformable objects. However, the reward function in reinforcement learning needs to measure the state configuration of the highly deformable object. Vision-based reward functions are difficult to implement, given the high dimensionality of the state and complex dynamic behavior. In this work, we propose the consideration of concepts beyond vision and incorporate other modalities which can be extracted from deformable objects. By integrating tactile sensor cells into a textile piece, proprioceptive capabilities are gained that are valuable as they provide a reward function to a reinforcement learning agent. We demonstrate on a low-cost dual robotic arm setup that a physical agent can learn on a single CPU core to fold a rectangular patch of textile in the real world based on a learned reward function from tactile information.

Sign in / Sign up

Export Citation Format

Share Document