scholarly journals Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

2020 ◽  
Vol 34 (06) ◽  
pp. 9794-9801
Author(s):  
Tomáš Brázdil ◽  
Krishnendu Chatterjee ◽  
Petr Novotný ◽  
Jiří Vahala
Keyword(s):  
Markov Decision Processes ◽  
Negative Impact ◽  
Optimization Criterion ◽  
Decision Processes ◽  
Risk Averse ◽  
Sequential Decision ◽  
Failure State ◽  
Markov Decision ◽  
Planning Algorithm ◽  
Low Probability

Markov decision processes (MDPs) are the defacto framework for sequential decision making in the presence of stochastic uncertainty. A classical optimization criterion for MDPs is to maximize the expected discounted-sum payoff, which ignores low probability catastrophic events with highly negative impact on the system. On the other hand, risk-averse policies require the probability of undesirable events to be below a given threshold, but they do not account for optimization of the expected payoff. We consider MDPs with discounted-sum payoff with failure states which represent catastrophic outcomes. The objective of risk-constrained planning is to maximize the expected discounted-sum payoff among risk-averse policies that ensure the probability to encounter a failure state is below a desired threshold. Our main contribution is an efficient risk-constrained planning algorithm that combines UCT-like search with a predictor learned through interaction with the MDP (in the style of AlphaZero) and with a risk-constrained action selection via linear programming. We demonstrate the effectiveness of our approach with experiments on classical MDPs from the literature, including benchmarks with an order of 106 states.

scholarly journals Expectation Optimization with Probabilistic Guarantees in POMDPs with Discounted-Sum Objectives

2018 ◽  
Author(s):  
Krishnendu Chatterjee ◽  
Adrián Elgyütt ◽  
Petr Novotný ◽  
Owen Rouillé
Keyword(s):  
Markov Decision Processes ◽  
Decision Processes ◽  
Decision Making Under Uncertainty ◽  
Risk Averse ◽  
Wide Range ◽  
Markov Decision ◽  
Expectation Optimization ◽  
Low Probability ◽  
Partially Observable ◽  
Standard Framework

Partially-observable Markov decision processes (POMDPs) with discounted-sum payoff are a standard framework to model a wide range of problems related to decision making under uncertainty. Traditionally, the goal has been to obtain policies that optimize the expectation of the discounted-sum payoff. A key drawback of the expectation measure is that even low probability events with extreme payoff can significantly affect the expectation, and thus the obtained policies are not necessarily risk averse. An alternate approach is to optimize the probability that the payoff is above a certain threshold, which allows to obtain risk-averse policies, but ignore optimization of the expectation. We consider the expectation optimization with probabilistic guarantee (EOPG) problem where the goal is to optimize the expectation ensuring that the payoff is above a given threshold with at least a specified probability. We present several results on the EOPG problem, including the first algorithm to solve it.

scholarly journals Enforcing Almost-Sure Reachability in POMDPs

2021 ◽  
pp. 602-625
Author(s):  
Sebastian Junges ◽  
Nils Jansen ◽  
Sanjit A. Seshia
Keyword(s):  
Markov Decision Processes ◽  
Empirical Evaluation ◽  
Decision Processes ◽  
Limited Information ◽  
Sequential Decision ◽  
Goal State ◽  
Learning Agent ◽  
Markov Decision ◽  
System Configurations ◽  
Partially Observable

AbstractPartially-Observable Markov Decision Processes (POMDPs) are a well-known stochastic model for sequential decision making under limited information. We consider the EXPTIME-hard problem of synthesising policies that almost-surely reach some goal state without ever visiting a bad state. In particular, we are interested in computing the winning region, that is, the set of system configurations from which a policy exists that satisfies the reachability specification. A direct application of such a winning region is the safe exploration of POMDPs by, for instance, restricting the behavior of a reinforcement learning agent to the region. We present two algorithms: A novel SAT-based iterative approach and a decision-diagram based alternative. The empirical evaluation demonstrates the feasibility and efficacy of the approaches.

scholarly journals Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes

2018 ◽  
Author(s):  
Shun Zhang ◽  
Edmund H. Durfee ◽  
Satinder Singh
Keyword(s):  
Side Effects ◽  
Markov Decision Processes ◽  
Decision Processes ◽  
Minimax Regret ◽  
Negative Side ◽  
Markov Decision ◽  
Planning Algorithm ◽  
Negative Side Effects

As it achieves a goal on behalf of its human user, an autonomous agent's actions may have side effects that change features of its environment in ways that negatively surprise its user. An agent that can be trusted to operate safely should thus only change features the user has explicitly permitted. We formalize this problem, and develop a planning algorithm that avoids potentially negative side effects given what the agent knows about (un)changeable features. Further, we formulate a provably minimax-regret querying strategy for the agent to selectively ask the user about features that it hasn't explicitly been told about. We empirically show how much faster it is than a more exhaustive approach and how much better its queries are than those found by the best known heuristic.

Quantile Markov Decision Processes

2021 ◽  
Author(s):  
Xiaocheng Li ◽  
Huaiyang Zhong ◽  
Margaret L. Brandeau
Keyword(s):  
Markov Decision Process ◽  
Markov Decision Processes ◽  
Decision Process ◽  
Value At Risk ◽  
Infinite Horizon ◽  
Decision Processes ◽  
Conditional Value At Risk ◽  
Sequential Decision ◽  
Optimal Drug ◽  
Markov Decision

Title: Sequential Decision Making Using Quantiles The goal of a traditional Markov decision process (MDP) is to maximize the expectation of cumulative reward over a finite or infinite horizon. In many applications, however, a decision maker may be interested in optimizing a specific quantile of the cumulative reward. For example, a physician may want to determine the optimal drug regime for a risk-averse patient with the objective of maximizing the 0.10 quantile of the cumulative reward; this is the cumulative improvement in health that is expected to occur with at least 90% probability for the patient. In “Quantile Markov Decision Processes,” X. Li, H. Zhong, and M. Brandeau provide analytic results to solve the quantile Markov decision process (QMDP) problem. They develop an efficient dynamic programming procedure that finds the optimal QMDP value function for all states and quantiles in one pass. The algorithm also extends to the MDP problem with a conditional value-at-risk objective.

scholarly journals Sensitivity Analysis in Markov Decision Processes with Uncertain Reward Parameters

2011 ◽  
Vol 48 (04) ◽  
pp. 954-967 ◽  
Author(s):  
Chin Hon Tan ◽  
Joseph C. Hartman
Keyword(s):  
Sensitivity Analysis ◽  
Markov Decision Processes ◽  
Lot Sizing ◽  
Optimal Solution ◽  
Decision Processes ◽  
Model Parameters ◽  
Sequential Decision ◽  
Estimation Errors ◽  
Bellman Equations ◽  
Markov Decision

Sequential decision problems can often be modeled as Markov decision processes. Classical solution approaches assume that the parameters of the model are known. However, model parameters are usually estimated and uncertain in practice. As a result, managers are often interested in how estimation errors affect the optimal solution. In this paper we illustrate how sensitivity analysis can be performed directly for a Markov decision process with uncertain reward parameters using the Bellman equations. In particular, we consider problems involving (i) a single stationary parameter, (ii) multiple stationary parameters, and (iii) multiple nonstationary parameters. We illustrate the applicability of this work through a capacitated stochastic lot-sizing problem.

scholarly journals Solving Transition Independent Decentralized Markov Decision Processes

2004 ◽  
Vol 22 ◽  
pp. 423-455 ◽  
Author(s):  
R. Becker ◽  
S. Zilberstein ◽  
V. Lesser ◽  
C. V. Goldman
Keyword(s):  
Markov Decision Processes ◽  
Optimal Algorithm ◽  
Decision Processes ◽  
Specific Class ◽  
Multi Agent Systems ◽  
Sequential Decision ◽  
Anytime Algorithm ◽  
Reward Function ◽  
Markov Decision ◽  
Multi Agent

Formal treatment of collaborative multi-agent systems has been lagging behind the rapid progress in sequential decision making by individual agents. Recent work in the area of decentralized Markov Decision Processes (MDPs) has contributed to closing this gap, but the computational complexity of these models remains a serious obstacle. To overcome this complexity barrier, we identify a specific class of decentralized MDPs in which the agents' transitions are independent. The class consists of independent collaborating agents that are tied together through a structured global reward function that depends on all of their histories of states and actions. We present a novel algorithm for solving this class of problems and examine its properties, both as an optimal algorithm and as an anytime algorithm. To our best knowledge, this is the first algorithm to optimally solve a non-trivial subclass of decentralized MDPs. It lays the foundation for further work in this area on both exact and approximate algorithms.

scholarly journals Sensitivity Analysis in Markov Decision Processes with Uncertain Reward Parameters

2011 ◽  
Vol 48 (4) ◽  
pp. 954-967 ◽  
Author(s):  
Chin Hon Tan ◽  
Joseph C. Hartman
Keyword(s):  
Sensitivity Analysis ◽  
Markov Decision Processes ◽  
Lot Sizing ◽  
Optimal Solution ◽  
Decision Processes ◽  
Model Parameters ◽  
Sequential Decision ◽  
Estimation Errors ◽  
Bellman Equations ◽  
Markov Decision

Sequential decision problems can often be modeled as Markov decision processes. Classical solution approaches assume that the parameters of the model are known. However, model parameters are usually estimated and uncertain in practice. As a result, managers are often interested in how estimation errors affect the optimal solution. In this paper we illustrate how sensitivity analysis can be performed directly for a Markov decision process with uncertain reward parameters using the Bellman equations. In particular, we consider problems involving (i) a single stationary parameter, (ii) multiple stationary parameters, and (iii) multiple nonstationary parameters. We illustrate the applicability of this work through a capacitated stochastic lot-sizing problem.

Sign in / Sign up

Export Citation Format

Share Document