Detecting optimal and non-optimal actions in average-cost Markov decision processes

1994 ◽  
Vol 31 (4) ◽  
pp. 979-990 ◽  
Author(s):  
Jean B. Lasserre
Keyword(s):  
Linear Programming ◽  
Markov Decision Processes ◽  
Average Cost ◽  
Sufficient Conditions ◽  
Iteration Scheme ◽  
Policy Iteration ◽  
Decision Processes ◽  
Ergodic Average ◽  
Linear Programming Methods ◽  
Markov Decision

We present two sufficient conditions for detection of optimal and non-optimal actions in (ergodic) average-cost MDPs. They are easily interpreted and can be implemented as detection tests in both policy iteration and linear programming methods. An efficient implementation of a recent new policy iteration scheme is discussed.

Detecting optimal and non-optimal actions in average-cost Markov decision processes

1994 ◽  
Vol 31 (04) ◽  
pp. 979-990
Author(s):  
Jean B. Lasserre
Keyword(s):  
Linear Programming ◽  
Markov Decision Processes ◽  
Average Cost ◽  
Sufficient Conditions ◽  
Iteration Scheme ◽  
Policy Iteration ◽  
Decision Processes ◽  
Ergodic Average ◽  
Linear Programming Methods ◽  
Markov Decision

We present two sufficient conditions for detection of optimal and non-optimal actions in (ergodic) average-cost MDPs. They are easily interpreted and can be implemented as detection tests in both policy iteration and linear programming methods. An efficient implementation of a recent new policy iteration scheme is discussed.

scholarly journals On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

2021 ◽  
Author(s):  
Huizhen Yu
Keyword(s):  
Linear Programming ◽  
Markov Decision Processes ◽  
Average Cost ◽  
Decision Processes ◽  
The State ◽  
Programming Approach ◽  
One Stage ◽  
Markov Decision ◽  
Action Spaces ◽  
Borel Measurable

We consider the linear programming approach for constrained and unconstrained Markov decision processes (MDPs) under the long-run average-cost criterion, where the class of MDPs in our study have Borel state spaces and discrete countable action spaces. Under a strict unboundedness condition on the one-stage costs and a recently introduced majorization condition on the state transition stochastic kernel, we study infinite-dimensional linear programs for the average-cost MDPs and prove the absence of a duality gap and other optimality results. Our results do not require a lower-semicontinuous MDP model. Thus, they can be applied to countable action space MDPs where the dynamics and one-stage costs are discontinuous in the state variable. Our proofs make use of the continuity property of Borel measurable functions asserted by Lusin’s theorem.

A new policy iteration scheme for Markov decision processes using Schweitzer's formula

1994 ◽  
Vol 31 (01) ◽  
pp. 268-273 ◽  
Author(s):  
J. B. Lasserre
Keyword(s):  
Markov Decision Processes ◽  
Exact Formula ◽  
Iteration Scheme ◽  
Policy Iteration ◽  
Decision Processes ◽  
Steady State Probability ◽  
Markov Decision ◽  
One Step ◽  
New Criterion ◽  
And Storage

Given a family of Markov chains with a single recurrent class, we present a potential application of Schweitzer's exact formula relating the steady-state probability and fundamental matrices of any two chains in the family. We propose a new policy iteration scheme for Markov decision processes where in contrast to policy iteration, the new criterion for selecting an action ensures the maximal one-step average cost improvement. Its computational complexity and storage requirement are analysed.

A new policy iteration scheme for Markov decision processes using Schweitzer's formula

1994 ◽  
Vol 31 (1) ◽  
pp. 268-273 ◽  
Author(s):  
J. B. Lasserre
Keyword(s):  
Markov Decision Processes ◽  
Exact Formula ◽  
Iteration Scheme ◽  
Policy Iteration ◽  
Decision Processes ◽  
Steady State Probability ◽  
Markov Decision ◽  
One Step ◽  
New Criterion ◽  
And Storage

Given a family of Markov chains with a single recurrent class, we present a potential application of Schweitzer's exact formula relating the steady-state probability and fundamental matrices of any two chains in the family. We propose a new policy iteration scheme for Markov decision processes where in contrast to policy iteration, the new criterion for selecting an action ensures the maximal one-step average cost improvement. Its computational complexity and storage requirement are analysed.

Extreme-point solutions in Markov decision processes

1983 ◽  
Vol 20 (04) ◽  
pp. 835-842
Author(s):  
David Assaf
Keyword(s):  
Convex Function ◽  
Extreme Point ◽  
Markov Decision Processes ◽  
Convex Functions ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Markov Decision ◽  
Full Solution

The paper presents sufficient conditions for certain functions to be convex. Functions of this type often appear in Markov decision processes, where their maximum is the solution of the problem. Since a convex function takes its maximum at an extreme point, the conditions may greatly simplify a problem. In some cases a full solution may be obtained after the reduction is made. Some illustrative examples are discussed.

scholarly journals Impulsive Control for Continuous-Time Markov Decision Processes

2015 ◽  
Vol 47 (1) ◽  
pp. 106-127 ◽  
Author(s):  
François Dufour ◽  
Alexei B. Piunovskiy
Keyword(s):  
Optimal Control ◽  
Control Problem ◽  
Markov Decision Processes ◽  
Control Strategy ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Optimal Control Strategy ◽  
Optimality Equation ◽  
Markov Decision

In this paper our objective is to study continuous-time Markov decision processes on a general Borel state space with both impulsive and continuous controls for the infinite time horizon discounted cost. The continuous-time controlled process is shown to be nonexplosive under appropriate hypotheses. The so-called Bellman equation associated to this control problem is studied. Sufficient conditions ensuring the existence and the uniqueness of a bounded measurable solution to this optimality equation are provided. Moreover, it is shown that the value function of the optimization problem under consideration satisfies this optimality equation. Sufficient conditions are also presented to ensure on the one hand the existence of an optimal control strategy, and on the other hand the existence of a ε-optimal control strategy. The decomposition of the state space into two disjoint subsets is exhibited where, roughly speaking, one should apply a gradual action or an impulsive action correspondingly to obtain an optimal or ε-optimal strategy. An interesting consequence of our previous results is as follows: the set of strategies that allow interventions at time t = 0 and only immediately after natural jumps is a sufficient set for the control problem under consideration.

Sign in / Sign up

Export Citation Format

Share Document