scholarly journals Hierarchical Reinforcement Learning Explains Task Interleaving Behavior

2020 ◽  
Author(s):  
Christoph Gebhardt ◽  
Antti Oulasvirta ◽  
Otmar Hilliges
Keyword(s):  
Reinforcement Learning ◽  
Supervisory Control ◽  
Question Answering ◽  
Greedy Heuristics ◽  
Marginal Rate ◽  
Individual Level ◽  
Cost Structures ◽  
Function Decomposition ◽  
Multiple Tasks ◽  
Decision To Stay

Abstract How do people decide how long to continue in a task, when to switch, and to which other task? It is known that task interleaving adapts situationally, showing sensitivity to changes in expected rewards, costs, and task boundaries. However, the mechanisms that underpin the decision to stay in a task versus switch away are not thoroughly understood. Previous work has explained task interleaving by greedy heuristics and a policy that maximizes the marginal rate of return. However, it is unclear how such a strategy would allow for adaptation to environments that offer multiple tasks with complex switch costs and delayed rewards. Here, we develop a hierarchical model of supervisory control driven by reinforcement learning (RL). The core assumption is that the supervisory level learns to switch using task-specific approximate utility estimates, which are computed on the lower level. We show that a hierarchically optimal value function decomposition can be learned from experience, even in conditions with multiple tasks and arbitrary and uncertain reward and cost structures. The model also reproduces well-known key phenomena of task interleaving, such as the sensitivity to costs of resumption and immediate as well as delayed in-task rewards. In a demanding task interleaving study with 211 human participants and realistic tasks (reading, mathematics, question-answering, recognition), the model yielded better predictions of individual-level data than a flat (non-hierarchical) RL model and an omniscient-myopic baseline. Corroborating emerging evidence from cognitive neuroscience, our results suggest hierarchical RL as a plausible model of supervisory control in task interleaving.

scholarly journals An Introduction to Intertask Transfer for Reinforcement Learning

AI Magazine ◽  
2011 ◽  
Vol 32 (1) ◽  
pp. 15 ◽  
Author(s):  
Matthew E. Taylor ◽  
Peter Stone
Keyword(s):  
Reinforcement Learning ◽  
Transfer Learning ◽  
Learning Problem ◽  
Open Problems ◽  
Learning Framework ◽  
Learning Domains ◽  
Multiple Tasks ◽  
Exciting Area ◽  
Generalize Information ◽  
Selection Of

Transfer learning has recently gained popularity due to the development of algorithms that can successfully generalize information across multiple tasks. This article focuses on transfer in the context of reinforcement learning domains, a general learning framework where an agent acts in an environment to maximize a reward signal. The goals of this article are to (1) familiarize readers with the transfer learning problem in reinforcement learning domains, (2) explain why the problem is both interesting and difficult, (3) present a selection of existing techniques that demonstrate different solutions, and (4) provide representative open problems in the hope of encouraging additional research in this exciting area.

scholarly journals Querying NoSQL with Deep Learning to Answer Natural Language Questions

2019 ◽  
Vol 33 ◽  
pp. 9416-9421
Author(s):  
Sebastian Blank ◽  
Florian Wilhelm ◽  
Hans-Peter Zorn ◽  
Achim Rettinger
Keyword(s):  
Deep Learning ◽  
Reinforcement Learning ◽  
Natural Language ◽  
Question Answering ◽  
Query Languages ◽  
Domain Specific ◽  
Nosql Database ◽  
End To End ◽  
Database Operations ◽  
Almost All

Almost all of today’s knowledge is stored in databases and thus can only be accessed with the help of domain specific query languages, strongly limiting the number of people which can access the data. In this work, we demonstrate an end-to-end trainable question answering (QA) system that allows a user to query an external NoSQL database by using natural language. A major challenge of such a system is the non-differentiability of database operations which we overcome by applying policy-based reinforcement learning. We evaluate our approach on Facebook’s bAbI Movie Dialog dataset and achieve a competitive score of 84.2% compared to several benchmark models. We conclude that our approach excels with regard to real-world scenarios where knowledge resides in external databases and intermediate labels are too costly to gather for non-end-to-end trainable QA systems.

scholarly journals MetaLight: Value-Based Meta-Reinforcement Learning for Traffic Signal Control

2020 ◽  
Vol 34 (01) ◽  
pp. 1153-1160 ◽  
Author(s):  
Xinshi Zang ◽  
Huaxiu Yao ◽  
Guanjie Zheng ◽  
Nan Xu ◽  
Kai Xu ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Real World ◽  
Learning Algorithm ◽  
Traffic Signal ◽  
Training Data ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Individual Level ◽  
Real World Datasets ◽  
Reinforcement Learning Models

Using reinforcement learning for traffic signal control has attracted increasing interests recently. Various value-based reinforcement learning methods have been proposed to deal with this classical transportation problem and achieved better performances compared with traditional transportation methods. However, current reinforcement learning models rely on tremendous training data and computational resources, which may have bad consequences (e.g., traffic jams or accidents) in the real world. In traffic signal control, some algorithms have been proposed to empower quick learning from scratch, but little attention is paid to learning by transferring and reusing learned experience. In this paper, we propose a novel framework, named as MetaLight, to speed up the learning process in new scenarios by leveraging the knowledge learned from existing scenarios. MetaLight is a value-based meta-reinforcement learning workflow based on the representative gradient-based meta-learning algorithm (MAML), which includes periodically alternate individual-level adaptation and global-level adaptation. Moreover, MetaLight improves the-state-of-the-art reinforcement learning model FRAP in traffic signal control by optimizing its model structure and updating paradigm. The experiments on four real-world datasets show that our proposed MetaLight not only adapts more quickly and stably in new traffic scenarios, but also achieves better performance.

scholarly journals Few-Shot Complex Knowledge Base Question Answering via Meta Reinforcement Learning

2020 ◽  
Author(s):  
Yuncheng Hua ◽  
Yuan-Fang Li ◽  
Gholamreza Haffari ◽  
Guilin Qi ◽  
Tongtong Wu
Keyword(s):  
Reinforcement Learning ◽  
Knowledge Base ◽  
Question Answering ◽  
Complex Knowledge

scholarly journals A Deep Reinforcement Learning Based Multi-Step Coarse to Fine Question Answering (MSCQA) System

2019 ◽  
Vol 33 ◽  
pp. 7224-7232 ◽  
Author(s):  
Yu Wang ◽  
Hongxia Jin
Keyword(s):  
Reinforcement Learning ◽  
Question Answering ◽  
State Of The Art ◽  
Learning Model ◽  
Current State ◽  
Coarse To Fine ◽  
Reinforcement Learning Model

In this paper, we present a multi-step coarse to fine question answering (MSCQA) system which can efficiently processes documents with different lengths by choosing appropriate actions. The system is designed using an actor-critic based deep reinforcement learning model to achieve multistep question answering. Compared to previous QA models targeting on datasets mainly containing either short or long documents, our multi-step coarse to fine model takes the merits from multiple system modules, which can handle both short and long documents. The system hence obtains a much better accuracy and faster trainings speed compared to the current state-of-the-art models. We test our model on four QA datasets, WIKEREADING, WIKIREADING LONG, CNN and SQuAD, and demonstrate 1.3%-1.7% accuracy improvements with 1.5x-3.4x training speed-ups in comparison to the baselines using state-of-the-art models.

scholarly journals A reinforcement learning formulation to the complex question answering problem

2015 ◽  
Vol 51 (3) ◽  
pp. 252-272 ◽  
Author(s):  
Yllias Chali ◽  
Sadid A. Hasan ◽  
Mustapha Mojahid
Keyword(s):  
Reinforcement Learning ◽  
Question Answering ◽  
Complex Question
Sign in / Sign up

Export Citation Format

Share Document