Perturbation theory for Markov reward processes with applications to queueing systems

1988 ◽  
Vol 20 (1) ◽  
pp. 79-98 ◽  
Author(s):  
Nico M. Van Dijk ◽  
Martin L. Puterman
Keyword(s):  
First Passage Time ◽  
Queueing Systems ◽  
Infinite Horizon ◽  
Passage Time ◽  
Decision Processes ◽  
Markov Decision ◽  
Reward Process ◽  
Reward Functions ◽  
Markov Reward ◽  
Time Bounds

We study the effect of perturbations in the data of a discrete-time Markov reward process on the finite-horizon total expected reward, the infinite-horizon expected discounted and average reward and the total expected reward up to a first-passage time. Bounds for the absolute errors of these reward functions are obtained. The results are illustrated for a finite as well as infinite queueing systems (M/M/1/S and ). Extensions to Markov decision processes and other settings are discussed.

Perturbation theory for Markov reward processes with applications to queueing systems

1988 ◽  
Vol 20 (01) ◽  
pp. 79-98 ◽  
Author(s):  
Nico M. Van Dijk ◽  
Martin L. Puterman
Keyword(s):  
First Passage Time ◽  
Queueing Systems ◽  
Infinite Horizon ◽  
Passage Time ◽  
Decision Processes ◽  
Markov Decision ◽  
Reward Process ◽  
Reward Functions ◽  
Markov Reward ◽  
Time Bounds

We study the effect of perturbations in the data of a discrete-time Markov reward process on the finite-horizon total expected reward, the infinite-horizon expected discounted and average reward and the total expected reward up to a first-passage time. Bounds for the absolute errors of these reward functions are obtained. The results are illustrated for a finite as well as infinite queueing systems (M/M/1/S and ). Extensions to Markov decision processes and other settings are discussed.

Functional Reward Markov Decision Processes: Theory and Applications

2017 ◽  
Vol 26 (03) ◽  
pp. 1760014
Author(s):  
Paul Weng ◽  
Olivier Spanjaard
Keyword(s):  
Markov Decision Processes ◽  
Infinite Horizon ◽  
Standard Form ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Markov Decision ◽  
Standard Models ◽  
Reward Functions ◽  
Planning Problems ◽  
Horizon Case

Markov decision processes (MDP) have become one of the standard models for decisiontheoretic planning problems under uncertainty. In its standard form, rewards are assumed to be numerical additive scalars. In this paper, we propose a generalization of this model allowing rewards to be functional. The value of a history is recursively computed by composing the reward functions. We show that several variants of MDPs presented in the literature can be instantiated in this setting. We then identify sufficient conditions on these reward functions for dynamic programming to be valid. We also discuss the infinite horizon case and the case where a maximum operator does not exist. In order to show the potential of our framework, we conclude the paper by presenting several illustrative examples.

Sign in / Sign up

Export Citation Format

Share Document