Vanishing discount approximations in controlled Markov chains with risk-sensitive average criterion

2018 ◽  
Vol 50 (01) ◽  
pp. 204-230 ◽  
Author(s):  
Rolando Cavazos-Cadena ◽  
Daniel Hernández-Hernández
Keyword(s):  
Average Cost ◽  
Control Policy ◽  
Sensitivity Coefficient ◽  
Value Functions ◽  
Risk Sensitive ◽  
Finite State ◽  
Markov Decision ◽  
Functional Part ◽  
Optimal Average ◽  
Average Cost Optimality Equation

Abstract This work concerns Markov decision chains on a finite state space. The decision-maker has a constant and nonnull risk sensitivity coefficient, and the performance of a control policy is measured by two different indices, namely, the discounted and average criteria. Motivated by well-known results for the risk-neutral case, the problem of approximating the optimal risk-sensitive average cost in terms of the optimal risk-sensitive discounted value functions is addressed. Under suitable communication assumptions, it is shown that, as the discount factor increases to 1, appropriate normalizations of the optimal discounted value functions converge to the optimal average cost, and to the functional part of the solution of the risk-sensitive average cost optimality equation.

The Computation of Average Optimal Policies in Denumerable State Markov Decision Chains

1997 ◽  
Vol 29 (01) ◽  
pp. 114-137
Author(s):  
Linn I. Sennott
Keyword(s):  
Discrete Time ◽  
Average Cost ◽  
Queueing Systems ◽  
State Spaces ◽  
Original Process ◽  
Optimal Policies ◽  
Finite State ◽  
Markov Decision ◽  
Optimal Average ◽  
Infinite State

This paper studies the expected average cost control problem for discrete-time Markov decision processes with denumerably infinite state spaces. A sequence of finite state space truncations is defined such that the average costs and average optimal policies in the sequence converge to the optimal average cost and an optimal policy in the original process. The theory is illustrated with several examples from the control of discrete-time queueing systems. Numerical results are discussed.

The Computation of Average Optimal Policies in Denumerable State Markov Decision Chains

1997 ◽  
Vol 29 (1) ◽  
pp. 114-137 ◽  
Author(s):  
Linn I. Sennott
Keyword(s):  
Discrete Time ◽  
Average Cost ◽  
Queueing Systems ◽  
State Spaces ◽  
Original Process ◽  
Optimal Policies ◽  
Finite State ◽  
Markov Decision ◽  
Optimal Average ◽  
Infinite State

This paper studies the expected average cost control problem for discrete-time Markov decision processes with denumerably infinite state spaces. A sequence of finite state space truncations is defined such that the average costs and average optimal policies in the sequence converge to the optimal average cost and an optimal policy in the original process. The theory is illustrated with several examples from the control of discrete-time queueing systems. Numerical results are discussed.

scholarly journals Explicit Solution of the Average-Cost Optimality Equation for a Pest-Control Problem

2011 ◽  
Vol 2011 ◽  
pp. 1-11
Author(s):  
Epaminondas G. Kyriakidis
Keyword(s):  
Optimal Control ◽  
Markov Decision Process ◽  
Pest Control ◽  
Continuous Time ◽  
Decision Process ◽  
Average Cost ◽  
Optimality Equation ◽  
Markov Decision ◽  
Average Cost Optimality Equation ◽  
Cost Optimality

We introduce a Markov decision process in continuous time for the optimal control of a simple symmetrical immigration-emigration process by the introduction of total catastrophes. It is proved that a particular control-limit policy is average cost optimal within the class of all stationary policies by verifying that the relative values of this policy are the solution of the corresponding optimality equation.

EXISTENCE OF OPTIMAL STATIONARY POLICIES IN FINITE DYNAMIC PROGRAMS WITH NONNEGATIVE REWARDS

2001 ◽  
Vol 15 (4) ◽  
pp. 557-564 ◽  
Author(s):  
Rolando Cavazos-Cadena ◽  
Raúl Montes-de-Oca
Keyword(s):  
Control Policy ◽  
Stationary Policy ◽  
Reward Function ◽  
Total Reward ◽  
Dynamic Programs ◽  
Finite State ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Action Spaces ◽  
Discounted Criterion

This article concerns Markov decision chains with finite state and action spaces, and a control policy is graded via the expected total-reward criterion associated to a nonnegative reward function. Within this framework, a classical theorem guarantees the existence of an optimal stationary policy whenever the optimal value function is finite, a result that is obtained via a limit process using the discounted criterion. The objective of this article is to present an alternative approach, based entirely on the properties of the expected total-reward index, to establish such an existence result.

scholarly journals Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

2005 ◽  
Vol 42 (4) ◽  
pp. 905-918 ◽  
Author(s):  
Rolando Cavazos-Cadena ◽  
Raúl Montes-De-Oca
Keyword(s):  
Iteration Algorithm ◽  
Value Iteration ◽  
Stationary Policy ◽  
Long Run ◽  
Risk Sensitive ◽  
Finite State ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Compact Action Sets ◽  
Nonstationary Value Iteration

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

Sign in / Sign up

Export Citation Format

Share Document