Technical Note—Cyclic Variables and Markov Decision Processes

Markov decision processes are commonly used to model forward-looking behavior. However, cyclic terms, including seasonality, are often omitted from these models because of the increase in computational burden. This paper develops a cyclic value function iteration (CVFI), an adjustment to the standard value function iteration. By updating states in a specific order, CVFI allows cyclic variables to be included in the state space with no increase in the computational cost. This result is proved theoretically and shown to hold closely in Monte Carlo simulations.

COMPUTING MARKOV-PERFECT OPTIMAL POLICIES IN BUSINESS-CYCLE MODELS

Time inconsistency is an essential feature of many policy problems. This paper presents and compares three methods for computing Markov-perfect optimal policies in stochastic nonlinear business cycle models. The methods considered include value function iteration, generalized Euler equations, and parameterized shadow prices. In the context of a business cycle model in which a fiscal authority chooses government spending and income taxation optimally, although lacking the ability to commit, we show that the solutions obtained using value function iteration and generalized Euler equations are somewhat more accurate than that obtained using parameterized shadow prices. Among these three methods, we show that value function iteration can be applied easily, even to environments that include a risk-sensitive fiscal authority and/or inequality constraints on government spending. We show that the risk-sensitive fiscal authority lowers government spending and income taxation, reducing the disincentive to accumulate wealth that households face.

Value Function Iteration as a Solution Method for the Ramsey Model

SummaryValue function iteration is one of the standard tools for the solution of dynamic general equilibrium models if the dimension of the state space is one ore two. We consider three kinds of models: the deterministic and the stochastic growth model and a simple heterogenous agent model. Each model is solved with six different algorithms: (1) simple value function iteration as compared to (2) smart value function iteration neglects the special structure of the problem. (3) Full and (4) modified policy iteration are methods to speed up convergence. (5) linear and (6) cubic interpolation between the grid points are methods that enhance precision and reduce the size of the grid. We evaluate the algorithms with respect to speed and accuracy. Accuracy is defined as the maximum absolute value of the residual of the Euler equation that determines the household’s savings. We demonstrate that the run time of all algorithms can be reduced substantially if the value function is initialized stepwise, starting on a coarse grid and increasing the number of grid points successively until the desired size is reached.We find that value function iteration with cubic spline interpolation between grid points dominates the other methods if a high level of accuracy is needed.