scholarly journals CONVERGENCE OF SIMULATION-BASED POLICY ITERATION

2003 ◽  
Vol 17 (2) ◽  
pp. 213-234 ◽  
Author(s):  
William L. Cooper ◽  
Shane G. Henderson ◽  
Mark E. Lewis
Keyword(s):  
Markov Decision Processes ◽  
Decision Rules ◽  
Almost Sure Convergence ◽  
Policy Iteration ◽  
Decision Processes ◽  
Optimal Decision ◽  
Iteration Algorithm ◽  
Simulation Based ◽  
Markov Decision ◽  
Run Lengths

Simulation-based policy iteration (SBPI) is a modification of the policy iteration algorithm for computing optimal policies for Markov decision processes. At each iteration, rather than solving the average evaluation equations, SBPI employs simulation to estimate a solution to these equations. For recurrent average-reward Markov decision processes with finite state and action spaces, we provide easily verifiable conditions that ensure that simulation-based policy iteration almost-surely eventually never leaves the set of optimal decision rules. We analyze three simulation estimators for solutions to the average evaluation equations. Using our general results, we derive simple conditions on the simulation run lengths that guarantee the almost-sure convergence of the algorithm.

scholarly journals Policy Iteration for Continuous-Time Average Reward Markov Decision Processes in Polish Spaces

2009 ◽  
Vol 2009 ◽  
pp. 1-17 ◽  
Author(s):  
Quanxin Zhu ◽  
Xinsong Yang ◽  
Chuangxia Huang
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Policy Iteration ◽  
Decision Processes ◽  
Iteration Algorithm ◽  
Average Reward ◽  
Stationary Policy ◽  
Optimality Equation ◽  
Markov Decision ◽  
Average Reward Optimality

We study thepolicy iteration algorithm(PIA) for continuous-time jump Markov decision processes in general state and action spaces. The corresponding transition rates are allowed to beunbounded, and the reward rates may haveneither upper nor lower bounds. The criterion that we are concerned with isexpected average reward. We propose a set of conditions under which we first establish the average reward optimality equation and present the PIA. Then under twoslightlydifferent sets of conditions we show that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation.

scholarly journals Stochastic Comparative Statics in Markov Decision Processes

2021 ◽  
Author(s):  
Bar Light
Keyword(s):  
Markov Decision Processes ◽  
Dynamic Pricing ◽  
Optimization Problems ◽  
Transition Probability ◽  
Random Variable ◽  
Comparative Statics ◽  
Decision Processes ◽  
Optimal Decision ◽  
Initial State ◽  
Markov Decision

In multiperiod stochastic optimization problems, the future optimal decision is a random variable whose distribution depends on the parameters of the optimization problem. I analyze how the expected value of this random variable changes as a function of the dynamic optimization parameters in the context of Markov decision processes. I call this analysis stochastic comparative statics. I derive both comparative statics results and stochastic comparative statics results showing how the current and future optimal decisions change in response to changes in the single-period payoff function, the discount factor, the initial state of the system, and the transition probability function. I apply my results to various models from the economics and operations research literature, including investment theory, dynamic pricing models, controlled random walks, and comparisons of stationary distributions.

Sign in / Sign up

Export Citation Format

Share Document