Dynamic Decision Networks Applications in Active Learning Simulators

Active learning simulators (ALSs) allow students to practice and carry out experiments in a safe environment – anytime, anywhere. Well-designed simulations may enhance learning, and provide the bridge from concept to practical understanding. Nevertheless, learning with ALS depends largely on the student’s ability to explore and interpret the performed experiments. By adding an Intelligent Tutoring System (ITS), it is possible to provide individualized personal guidance to students. The challenges are how an ITS properly assesses the cognitive state of the student based on the results of experiments and the student’s interaction, and how it provides adaptive feedback to the student. In this chapter we describe how an ITS based on Dynamic Decision Networks (DDNs) is applied in an undergraduate Physics scenario where the aim is to adapt the learning experience to suit the learners’ needs. We propose employing Probabilistic Relational Models (PRMs) to facilitate the construction of the model. These are frameworks that enable the definition of Probabilistic Graphical and Entity Relationship Models, starting from a domain, and in this case, environments of ALSs. With this representation, the tutor can be easily adapted to different experiments, domains, and student levels, thereby minimizing the development effort for building and integrating Intelligent Tutoring Systems (ITS) for ALSs. A discussion of the methodology is addressed, and preliminary results are presented.

Introduction

This chapter gives a general introduction to decision-theoretic models in artificial intelligence and an overview of the rest of the book. It starts by motivating the use of decision-theoretic models in artificial intelligence and discussing the challenges that arise as these techniques are applied to develop intelligent systems for complex domains. Then it introduces decision theory, including its axiomatic bases and the principle of maximum expected utility; a brief introduction to decision trees is also presented. Finally, an overview of the three main parts of the book –fundamentals, concepts and solutions– is presented.

Dynamic LIMIDS

One of the objectives of artificial intelligence is to build decision-support models for systems that evolve over time and include several types of uncertainty. Dynamic limited-memory influence diagrams (DLIMIDs) are a new type of model proposed recently for this kind of problems. DLIMIDs are similar to other models in assuming a multi-stage process that satisfies the Markov property, i.e., that the future is independent of the past given the current state. The main difference with those models is the restriction of limited memory, which means that the decision maker must make a choice based only on recent observations, but this limitation can be circumvented by the use of memory variables. We present several algorithms for evaluating DLIMIDs, show a real-world model for a medical problem, and compare DLIMIDs with related formalisms, such as LIMIDs, dynamic influence diagrams, and POMDPs.

Inference Strategies for Solving Semi-Markov Decision Processes

Semi-Markov decision processes are used to formulate many control problems and also play a key role in hierarchical reinforcement learning. In this chapter we show how to translate the decision making problem into a form that can instead be solved by inference and learning techniques. In particular, we will establish a formal connection between planning in semi-Markov decision processes and inference in probabilistic graphical models, then build on this connection to develop an expectation maximization (EM) algorithm for policy optimization in these models.

POMDP Models for Assistive Technology

This chapter presents a general decision theoretic model of interactions between users and cognitive assistive technologies for various tasks of importance to the elderly population. The model is a partially observable Markov decision process (POMDP) whose goal is to work in conjunction with a user towards the completion of a given activity or task. This requires the model to monitor and assist the user, to maintain indicators of overall user health, and to adapt to changes. The key strengths of the POMDP model are that it is able to deal with uncertainty, it is easy to specify, it can be applied to different tasks with little modification, and it is able to learn and adapt to changing tasks and situations. This chapter describes the model, gives a general learning method which enables the model to be learned from partially labeled data, and shows how the model can be applied within our research program on technologies for wellness. In particular, we show how the model is used in four tasks: assisted handwashing, stroke rehabilitation, health and safety monitoring, and wheelchair mobility. The first two have been fully implemented and tested, and results are summarized. The second two are meant to demonstrate how the POMDP can be applied across a wide variety of domains, but do not have specific implementations or results. The chapter gives an overview of ongoing work into each of these areas, and discusses future directions.

Applications of DEC-MDPs in Multi-Robot Systems

In this chapter, we introduce problematics related to the decentralized control of multi-robot systems. We first describe some applicative domains and review the main characteristics of the decision problems the robots must deal with. Then, we review some existing approaches to solve problems of multiagent decentralized control in stochastic environments. We present the Decentralized Markov Decision Processes and discuss their applicability to real-world multi-robot applications. Then, we introduce OC-DEC-MDPs and 2V-DEC-MDPs which have been developed to increase the applicability of DEC-MDPs.

Task Coordination for Service Robots Based on Multiple Markov Decision Processes

Markov Decision Processes (MDPs) provide a principled framework for planing under uncertainty. However, in general they assume a single action per decision epoch. In service robot applications, multiple tasks are required simultaneously, such as navigation, localization and interaction. We have developed a novel framework based on functional decomposition that divides a complex problem into several sub-problems. Each sub-problem is defined as an MDP and solved independently, and their individual policies are combined to obtain a global policy. In contrast to most previous approaches for hierarchical MDPs, in our approach all the MDPs work in parallel, so we obtain a reactive system based on a decision theoretic framework. We initially solved each MDP independently and combined their policies assuming no conflicts. Then we defined two kinds of conflicts, resource and behavior conflicts, and proposed solutions for both. The first kind of conflict is solved off-line using a two phase process which guarantees a near-optimal global policy. Behavior conflicts are solved on-line based on a set of restrictions specified by the user, and a constraint satisfaction module that selects the action set with higher expected utility. We have used these methods for task coordination in service robots, and present experimental results for a messenger robot.

A Case Study of Applying Decision Theory in the Real World

Spoken dialog systems present a classic example of planning under uncertainty. Speech recognition errors are ubiquitous and impossible to detect reliably, so the state of the conversation can never be known with certainty. Despite this, the system must choose actions to make progress to a long term goal. As such, decision theory, and in particular partially-observable Markov decision processes (POMDPs), present an attractive approach to building spoken dialog systems. Initial work on “toy” dialog systems validated the benefits of the POMDP approach; however, it also found that straightforward application of POMDPs could not scale to real-world problems. Subsequent work by a number of research teams has scaled up planning and belief monitoring, incorporated high-fidelity user simulations, and married commercial development practices with automatic optimization. Today, statistical dialog systems are being fielded by research labs for public use. This chapter traces the history of POMDP-based spoken dialog systems, and sketches avenues for future work.

An Intelligent Assistant for Power Plant Operation and Training Based on Decision-Theoretic Planning

In this chapter we present AsistO, a simulation-based intelligent assistant for power plant operators that provides on-line guidance in the form of ordered recommendations. These recommendations are generated using the formalism of Markov decision processes over an approximated factored representation of the plant. The decision model approximation is based on machine learning tools. We also described an explanation mechanism over these recommendations based on i) the selection of a relevant variable and ii) the automated construction of graphical explanations for operators. The explanation module analyzes the recommender system’s decision model to support the reason why a recommendation should be followed.

Relational Representations and Traces for Efficient Reinforcement Learning

This chapter introduces an approach for reinforcement learning based on a relational representation that: (i) can be applied over large search spaces, (ii) can incorporate domain knowledge, and (iii) can use previously learned policies on different, but similar, problems. The underlying idea is to represent states as sets of first order relations, actions in terms of those relations, and to learn policies over such generalized representation. It is shown how this representation can produce powerful abstractions and that policies learned over this generalized representation can be directly applied, without any further learning, to other problems that can be characterized by the same set of relations. To accelerate the learning process, we present an extension where traces of the tasks to be learned are provided by the user. These traces are used to select only a small subset of possible actions increasing the convergence of the learning algorithms. The effectiveness of the approach is tested on a flight simulator and on a mobile robot.