First Passage Optimality for Continuous-Time Markov Decision Processes With Varying Discount Factors and History-Dependent Policies

2014 ◽  
Vol 59 (1) ◽  
pp. 163-174 ◽  
Author(s):  
Xianping Guo ◽  
Xinyuan Song ◽  
Yi Zhang
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Decision Processes ◽  
First Passage ◽  
Discount Factors ◽  
Markov Decision

scholarly journals First Passage Optimality and Variance Minimisation of Markov Decision Processes with Varying Discount Factors

2015 ◽  
Vol 52 (2) ◽  
pp. 441-456 ◽  
Author(s):  
Xiao Wu ◽  
Xianping Guo
Keyword(s):  
Markov Decision Processes ◽  
Discrete Time ◽  
Infinite Horizon ◽  
Decision Processes ◽  
Previous Literature ◽  
First Passage ◽  
Discount Factors ◽  
Markov Decision ◽  
The Difference ◽  
Minimisation Problem

This paper deals with the first passage optimality and variance minimisation problems of discrete-time Markov decision processes (MDPs) with varying discount factors and unbounded rewards/costs. First, under suitable conditions slightly weaker than those in the previous literature on the standard (infinite horizon) discounted MDPs, we establish the existence and characterisation of the first passage expected-optimal stationary policies. Second, to further distinguish the expected-optimal stationary policies, we introduce the variance minimisation problem, prove that it is equivalent to a new first passage optimality problem of MDPs, and, thus, show the existence of a variance-optimal policy that minimises the variance over the set of all first passage expected-optimal stationary policies. Finally, we use a computable example to illustrate our main results and also to show the difference between the first passage optimality here and the standard discount optimality of MDPs in the previous literature.

First Passage Optimality and Variance Minimisation of Markov Decision Processes with Varying Discount Factors

2015 ◽  
Vol 52 (02) ◽  
pp. 441-456 ◽  
Author(s):  
Xiao Wu ◽  
Xianping Guo
Keyword(s):  
Markov Decision Processes ◽  
Discrete Time ◽  
Infinite Horizon ◽  
Decision Processes ◽  
Previous Literature ◽  
First Passage ◽  
Discount Factors ◽  
Markov Decision ◽  
The Difference ◽  
Minimisation Problem

This paper deals with the first passage optimality and variance minimisation problems of discrete-time Markov decision processes (MDPs) with varying discount factors and unbounded rewards/costs. First, under suitable conditions slightly weaker than those in the previous literature on the standard (infinite horizon) discounted MDPs, we establish the existence and characterisation of the first passage expected-optimal stationary policies. Second, to further distinguish the expected-optimal stationary policies, we introduce the variance minimisation problem, prove that it is equivalent to a new first passage optimality problem of MDPs, and, thus, show the existence of a variance-optimal policy that minimises the variance over the set of all first passage expected-optimal stationary policies. Finally, we use a computable example to illustrate our main results and also to show the difference between the first passage optimality here and the standard discount optimality of MDPs in the previous literature.

scholarly journals Denumerable state continuous time Markov decision processes with unbounded cost and transition rates under average criterion

2002 ◽  
Vol 43 (4) ◽  
pp. 541-557 ◽  
Author(s):  
Xianping Guo ◽  
Weiping Zhu
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Decision Processes ◽  
Transition Rates ◽  
Birth And Death Processes ◽  
Optimality Equation ◽  
Average Criterion ◽  
Markov Decision ◽  
Unbounded Cost ◽  
Queue Model

AbstractIn this paper, we consider denumerable state continuous time Markov decision processes with (possibly unbounded) transition and cost rates under average criterion. We present a set of conditions and prove the existence of both average cost optimal stationary policies and a solution of the average optimality equation under the conditions. The results in this paper are applied to an admission control queue model and controlled birth and death processes.

Sign in / Sign up

Export Citation Format

Share Document