Conditions for the uniqueness of optimal policies of discounted Markov decision processes

2004 ◽  
Vol 60 (3) ◽  
pp. 415-436 ◽  
Author(s):  
Daniel Cruz-Su�rez ◽  
Ra�l Montes-de-Oca ◽  
Francisco Salem-Silva
Keyword(s):  
Markov Decision Processes ◽  
Decision Processes ◽  
Optimal Policies ◽  
Markov Decision

Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model

1983 ◽  
Vol 15 (2) ◽  
pp. 274-303 ◽  
Author(s):  
Arie Hordijk ◽  
Frank A. Van Der Duyn Schouten
Keyword(s):  
Markov Decision Processes ◽  
Optimal Policy ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Time Parameter ◽  
Queueing Model ◽  
Replacement Model ◽  
Optimal Policies ◽  
Markov Decision

Recently the authors introduced the concept of Markov decision drift processes. A Markov decision drift process can be seen as a straightforward generalization of a Markov decision process with continuous time parameter. In this paper we investigate the existence of stationary average optimal policies for Markov decision drift processes. Using a well-known Abelian theorem we derive sufficient conditions, which guarantee that a ‘limit point' of a sequence of discounted optimal policies with the discounting factor approaching 1 is an average optimal policy. An alternative set of sufficient conditions is obtained for the case in which the discounted optimal policies generate regenerative stochastic processes. The latter set of conditions is easier to verify in several applications. The results of this paper are also applicable to Markov decision processes with discrete or continuous time parameter and to semi-Markov decision processes. In this sense they generalize some well-known results for Markov decision processes with finite or compact action space. Applications to an M/M/1 queueing model and a maintenance replacement model are given. It is shown that under certain conditions on the model parameters the average optimal policy for the M/M/1 queueing model is monotone non-decreasing (as a function of the number of waiting customers) with respect to the service intensity and monotone non-increasing with respect to the arrival intensity. For the maintenance replacement model we prove the average optimality of a bang-bang type policy. Special attention is paid to the computation of the optimal control parameters.

Cost rate heuristics for semi-Markov decision processes

1992 ◽  
Vol 29 (03) ◽  
pp. 633-644
Author(s):  
K. D. Glazebrook ◽  
Michael P. Bailey ◽  
Lyn R. Whitaker
Keyword(s):  
Dynamic Programming ◽  
Markov Decision Processes ◽  
Preventive Maintenance ◽  
Decision Processes ◽  
Cost Rate ◽  
Backwards Induction ◽  
Optimal Policies ◽  
Markov Decision ◽  
Speed Of Evolution

In response to the computational complexity of the dynamic programming/backwards induction approach to the development of optimal policies for semi-Markov decision processes, we propose a class of heuristics resulting from an inductive process which proceeds forwards in time. These heuristics always choose actions in such a way as to minimize some measure of the current cost rate. We describe a procedure for calculating such cost rate heuristics. The quality of the performance of such policies is related to the speed of evolution (in a cost sense) of the process. A simple model of preventive maintenance is described in detail. Cost rate heuristics for this problem are calculated and assessed computationally.

Determining the Best Trade-Off Between Expected Economic Return and the Risk of Undesirable Events When Managing a Randomly Varying Population

1979 ◽  
Vol 36 (8) ◽  
pp. 939-947 ◽  
Author(s):  
Roy Mendelssohn
Keyword(s):  
Markov Decision Processes ◽  
Population Based ◽  
Decision Processes ◽  
Expected Return ◽  
Trade Off ◽  
Long Run ◽  
Optimal Policies ◽  
Markov Decision ◽  
Long Run Risk ◽  
Varying Population

Conditions are given that imply there exist policies that "minimize risk" of undesirable events for stochastic harvesting models. It is shown that for many problems, either such a policy will not exist, or else it is an "extreme" policy that is equally undesirable. Techniques are given to systematically trade-off decreases in the long-run expected return with decreases in the long-run risk. Several numerical examples are given for models of salmon runs, when both population-based risks and harvest-based risks are considered. Key words: Markov decision processes, risk, salmon management, Pareto optimal policies, trade-off curves, linear programing

scholarly journals Optimal policies for constrained average-cost Markov decision processes

Top ◽  
2009 ◽  
Vol 19 (1) ◽  
pp. 107-120 ◽  
Author(s):  
Juan González-Hernández ◽  
César E. Villarreal
Keyword(s):  
Markov Decision Processes ◽  
Average Cost ◽  
Decision Processes ◽  
Optimal Policies ◽  
Markov Decision

Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model

1983 ◽  
Vol 15 (02) ◽  
pp. 274-303 ◽  
Author(s):  
Arie Hordijk ◽  
Frank A. Van Der Duyn Schouten
Keyword(s):  
Markov Decision Processes ◽  
Optimal Policy ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Time Parameter ◽  
Queueing Model ◽  
Replacement Model ◽  
Optimal Policies ◽  
Markov Decision

Recently the authors introduced the concept of Markov decision drift processes. A Markov decision drift process can be seen as a straightforward generalization of a Markov decision process with continuous time parameter. In this paper we investigate the existence of stationary average optimal policies for Markov decision drift processes. Using a well-known Abelian theorem we derive sufficient conditions, which guarantee that a ‘limit point' of a sequence of discounted optimal policies with the discounting factor approaching 1 is an average optimal policy. An alternative set of sufficient conditions is obtained for the case in which the discounted optimal policies generate regenerative stochastic processes. The latter set of conditions is easier to verify in several applications. The results of this paper are also applicable to Markov decision processes with discrete or continuous time parameter and to semi-Markov decision processes. In this sense they generalize some well-known results for Markov decision processes with finite or compact action space. Applications to an M/M/1 queueing model and a maintenance replacement model are given. It is shown that under certain conditions on the model parameters the average optimal policy for the M/M/1 queueing model is monotone non-decreasing (as a function of the number of waiting customers) with respect to the service intensity and monotone non-increasing with respect to the arrival intensity. For the maintenance replacement model we prove the average optimality of a bang-bang type policy. Special attention is paid to the computation of the optimal control parameters.

Sign in / Sign up

Export Citation Format

Share Document