On Structural Properties of Optimal Average Cost functions in Markov Decision Processes with Borel Spaces and Universally Measurable Policies

2021 ◽  
pp. 125954
Author(s):  
Huizhen Yu
Keyword(s):  
Structural Properties ◽  
Markov Decision Processes ◽  
Average Cost ◽  
Decision Processes ◽  
Cost Functions ◽  
Markov Decision ◽  
Optimal Average

scholarly journals New discount and average optimality conditions for continuous-time Markov decision processes

2010 ◽  
Vol 42 (04) ◽  
pp. 953-985 ◽  
Author(s):  
Xianping Guo ◽  
Liuer Ye
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Average Cost ◽  
Nonnegative Solution ◽  
Decision Processes ◽  
Stationary Policy ◽  
Discounted Cost ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Bounded Below

This paper deals with continuous-time Markov decision processes in Polish spaces, under the discounted and average cost criteria. All underlying Markov processes are determined by given transition rates which are allowed to be unbounded, and the costs are assumed to be bounded below. By introducing an occupation measure of a randomized Markov policy and analyzing properties of occupation measures, we first show that the family of all randomized stationary policies is ‘sufficient’ within the class of all randomized Markov policies. Then, under the semicontinuity and compactness conditions, we prove the existence of a discounted cost optimal stationary policy by providing a value iteration technique. Moreover, by developing a new average cost, minimum nonnegative solution method, we prove the existence of an average cost optimal stationary policy under some reasonably mild conditions. Finally, we use some examples to illustrate applications of our results. Except that the costs are assumed to be bounded below, the conditions for the existence of discounted cost (or average cost) optimal policies are much weaker than those in the previous literature, and the minimum nonnegative solution approach is new.

scholarly journals New discount and average optimality conditions for continuous-time Markov decision processes

2010 ◽  
Vol 42 (4) ◽  
pp. 953-985 ◽  
Author(s):  
Xianping Guo ◽  
Liuer Ye
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Average Cost ◽  
Nonnegative Solution ◽  
Decision Processes ◽  
Stationary Policy ◽  
Discounted Cost ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Bounded Below

This paper deals with continuous-time Markov decision processes in Polish spaces, under the discounted and average cost criteria. All underlying Markov processes are determined by given transition rates which are allowed to be unbounded, and the costs are assumed to be bounded below. By introducing an occupation measure of a randomized Markov policy and analyzing properties of occupation measures, we first show that the family of all randomized stationary policies is ‘sufficient’ within the class of all randomized Markov policies. Then, under the semicontinuity and compactness conditions, we prove the existence of a discounted cost optimal stationary policy by providing a value iteration technique. Moreover, by developing a new average cost, minimum nonnegative solution method, we prove the existence of an average cost optimal stationary policy under some reasonably mild conditions. Finally, we use some examples to illustrate applications of our results. Except that the costs are assumed to be bounded below, the conditions for the existence of discounted cost (or average cost) optimal policies are much weaker than those in the previous literature, and the minimum nonnegative solution approach is new.

Average Cost Semi-Markov Decision Processes and the Control of Queueing Systems

1989 ◽  
Vol 3 (2) ◽  
pp. 247-272 ◽  
Author(s):  
Linn I. Sennott
Keyword(s):  
Markov Decision Processes ◽  
Average Cost ◽  
Queueing Systems ◽  
Decision Processes ◽  
Single Server ◽  
Stationary Policy ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Poisson Arrivals ◽  
Action Spaces

Semi-Markov decision processes underlie the control of many queueing systems. In this paper, we deal with infinite state semi-Markov decision processes with nonnegative, unbounded costs and finite action sets. Axioms for the existence of an expected average cost optimal stationary policy are presented. These conditions generalize the work in Sennott [22] for Markov decision processes. Verifiable conditions for the axioms to hold are obtained. The theory is applied to control of the M/G/l queue with variable service parameter, with on-off server, and with batch processing, and to control of the G/M/m queue with variable arrival parameter and customer rejection. It is applied to a timesharing network of queues with a single server and finally to optimal routing of Poisson arrivals to parallel exponential servers. The final section extends the existence result to compact action spaces.

Sign in / Sign up

Export Citation Format

Share Document