Performance Optimization of Semi-Markov Decision Processes with Discounted-cost Criteria

2008 ◽  
Vol 14 (3) ◽  
pp. 213-222 ◽  
Author(s):  
Baoqun Yin ◽  
Yanjie Li ◽  
Yaping Zhou ◽  
Hongsheng Xi
Keyword(s):  
Markov Decision Processes ◽  
Performance Optimization ◽  
Decision Processes ◽  
Discounted Cost ◽  
Markov Decision

scholarly journals New discount and average optimality conditions for continuous-time Markov decision processes

2010 ◽  
Vol 42 (04) ◽  
pp. 953-985 ◽  
Author(s):  
Xianping Guo ◽  
Liuer Ye
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Average Cost ◽  
Nonnegative Solution ◽  
Decision Processes ◽  
Stationary Policy ◽  
Discounted Cost ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Bounded Below

This paper deals with continuous-time Markov decision processes in Polish spaces, under the discounted and average cost criteria. All underlying Markov processes are determined by given transition rates which are allowed to be unbounded, and the costs are assumed to be bounded below. By introducing an occupation measure of a randomized Markov policy and analyzing properties of occupation measures, we first show that the family of all randomized stationary policies is ‘sufficient’ within the class of all randomized Markov policies. Then, under the semicontinuity and compactness conditions, we prove the existence of a discounted cost optimal stationary policy by providing a value iteration technique. Moreover, by developing a new average cost, minimum nonnegative solution method, we prove the existence of an average cost optimal stationary policy under some reasonably mild conditions. Finally, we use some examples to illustrate applications of our results. Except that the costs are assumed to be bounded below, the conditions for the existence of discounted cost (or average cost) optimal policies are much weaker than those in the previous literature, and the minimum nonnegative solution approach is new.

scholarly journals New discount and average optimality conditions for continuous-time Markov decision processes

2010 ◽  
Vol 42 (4) ◽  
pp. 953-985 ◽  
Author(s):  
Xianping Guo ◽  
Liuer Ye
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Average Cost ◽  
Nonnegative Solution ◽  
Decision Processes ◽  
Stationary Policy ◽  
Discounted Cost ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Bounded Below

This paper deals with continuous-time Markov decision processes in Polish spaces, under the discounted and average cost criteria. All underlying Markov processes are determined by given transition rates which are allowed to be unbounded, and the costs are assumed to be bounded below. By introducing an occupation measure of a randomized Markov policy and analyzing properties of occupation measures, we first show that the family of all randomized stationary policies is ‘sufficient’ within the class of all randomized Markov policies. Then, under the semicontinuity and compactness conditions, we prove the existence of a discounted cost optimal stationary policy by providing a value iteration technique. Moreover, by developing a new average cost, minimum nonnegative solution method, we prove the existence of an average cost optimal stationary policy under some reasonably mild conditions. Finally, we use some examples to illustrate applications of our results. Except that the costs are assumed to be bounded below, the conditions for the existence of discounted cost (or average cost) optimal policies are much weaker than those in the previous literature, and the minimum nonnegative solution approach is new.

Discounted Cost Markov Decision Processes with a Constraint

1998 ◽  
Vol 12 (2) ◽  
pp. 177-187 ◽  
Author(s):  
Kazuyoshi Wakuta
Keyword(s):  
Markov Decision Process ◽  
Markov Decision Processes ◽  
Decision Process ◽  
Decision Processes ◽  
Discounted Cost ◽  
Markov Decision ◽  
Vector Valued

We consider a discounted cost Markov decision process with a constraint. Relating this to a vector-valued Markov decision process, we prove that there exists a constrained optimal randomized semistationary policy if there exists at least one policy satisfying a constraint. Moreover, we present an algorithm by which we can find the constrained optimal randomized semistationary policy, or we can discover that there exist no policies satisfying a given constraint.

PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES AND PERIODIC POLICIES WITH APPLICATIONS

2011 ◽  
Vol 10 (06) ◽  
pp. 1175-1197 ◽  
Author(s):  
JOHN GOULIONIS ◽  
D. STENGOS
Keyword(s):  
Markov Decision Processes ◽  
Piecewise Linear ◽  
Linear Equations ◽  
Infinite Horizon ◽  
Decision Processes ◽  
Value Functions ◽  
Discounted Cost ◽  
Markov Decision ◽  
Partially Observable Markov ◽  
Partially Observable

This paper treats the infinite horizon discounted cost control problem for partially observable Markov decision processes. Sondik studied the class of finitely transient policies and showed that their value functions over an infinite time horizon are piecewise linear (p.w.l) and can be computed exactly by solving a system of linear equations. However, the condition for finite transience is stronger than is needed to ensure p.w.l. value functions. In this paper, we introduce alternatively the class of periodic policies whose value functions turn out to be also p.w.l. Moreover, we examine a more general condition than finite transience and periodicity that ensures p.w.l. value functions. We implement these ideas in a replacement problem under Markovian deterioration, investigate for periodic policies and give numerical examples.

Sign in / Sign up

Export Citation Format

Share Document