How Does the Value Function of a Markov Decision Process Depend on the Transition Probabilities?

1997 ◽  
Vol 22 (4) ◽  
pp. 872-885 ◽  
Author(s):  
Alfred Müller
Keyword(s):  
Markov Decision Process ◽  
Decision Process ◽  
Value Function ◽  
Transition Probabilities ◽  
Markov Decision ◽  
The Value Function

On Optimal Terminal Wealth Problems with Random Trading Times and Drawdown Constraints

2014 ◽  
Vol 46 (01) ◽  
pp. 121-138 ◽  
Author(s):  
Ulrich Rieder ◽  
Marc Wittlinger
Keyword(s):  
Markov Decision Process ◽  
Decision Process ◽  
Value Function ◽  
Stationary Policy ◽  
General Utility ◽  
Investment Problem ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Running Maximum ◽  
The Value Function

We consider an investment problem where observing and trading are only possible at random times. In addition, we introduce drawdown constraints which require that the investor's wealth does not fall under a prior fixed percentage of its running maximum. The financial market consists of a riskless bond and a stock which is driven by a Lévy process. Moreover, a general utility function is assumed. In this setting we solve the investment problem using a related limsup Markov decision process. We show that the value function can be characterized as the unique fixed point of the Bellman equation and verify the existence of an optimal stationary policy. Under some mild assumptions the value function can be approximated by the value function of a contracting Markov decision process. We are able to use Howard's policy improvement algorithm for computing the value function as well as an optimal policy. These results are illustrated in a numerical example.

On Optimal Terminal Wealth Problems with Random Trading Times and Drawdown Constraints

2014 ◽  
Vol 46 (1) ◽  
pp. 121-138 ◽  
Author(s):  
Ulrich Rieder ◽  
Marc Wittlinger
Keyword(s):  
Markov Decision Process ◽  
Decision Process ◽  
Value Function ◽  
Stationary Policy ◽  
General Utility ◽  
Investment Problem ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Running Maximum ◽  
The Value Function

We consider an investment problem where observing and trading are only possible at random times. In addition, we introduce drawdown constraints which require that the investor's wealth does not fall under a prior fixed percentage of its running maximum. The financial market consists of a riskless bond and a stock which is driven by a Lévy process. Moreover, a general utility function is assumed. In this setting we solve the investment problem using a related limsup Markov decision process. We show that the value function can be characterized as the unique fixed point of the Bellman equation and verify the existence of an optimal stationary policy. Under some mild assumptions the value function can be approximated by the value function of a contracting Markov decision process. We are able to use Howard's policy improvement algorithm for computing the value function as well as an optimal policy. These results are illustrated in a numerical example.

Optimal Occupation in the Complete Graph

1993 ◽  
Vol 7 (3) ◽  
pp. 369-385 ◽  
Author(s):  
Kyle Siegrist
Keyword(s):  
Markov Decision Process ◽  
Complete Graph ◽  
Decision Process ◽  
Value Function ◽  
Comparison Result ◽  
State Action ◽  
Optimal Policies ◽  
Markov Decision ◽  
The Cost ◽  
The Value Function

We consider N sites (N ≤ ∞), each of which may be either occupied or unoccupied. Time is discrete, and at each time unit a set of occupied sites may attempt to capture a previously unoccupied site. The attempt will be successful with a probability that depends on the number of sites making the attempt, in which case the new site will also be occupied. A benefit is gained when new sites are occupied, but capture attempts are costly. The problem of optimal occupation is formulated as a Markov decision process in which the admissible actions are occupation strategies and the cost is a function of the strategy and the number of occupied sites. A partial order on the state-action pairs is used to obtain a comparison result for stationary policies and qualitative results concerning monotonicity of the value function for the n-stage problem (n ≤ ∞). The optimal policies are partially characterized when the cost depends on the action only through the total number of occupation attempts made.

scholarly journals An Offloading Algorithm based on Markov Decision Process in Mobile Edge Computing System

2022 ◽  
Vol 16 ◽  
pp. 115-121
Author(s):  
Bingxin Yao ◽  
Bin Wu ◽  
Siyun Wu ◽  
Yin Ji ◽  
Danggui Chen ◽  
...  
Keyword(s):  
Energy Consumption ◽  
Markov Decision Process ◽  
Decision Process ◽  
Value Function ◽  
Wireless Channel ◽  
Edge Computing ◽  
Iteration Algorithm ◽  
Mobile Edge Computing ◽  
Markov Decision ◽  
The Value Function

In this paper, an offloading algorithm based on Markov Decision Process (MDP) is proposed to solve the multi-objective offloading decision problem in Mobile Edge Computing (MEC) system. The feature of the algorithm is that MDP is used to make offloading decision. The number of tasks in the task queue, the number of accessible edge clouds and Signal-Noise-Ratio (SNR) of the wireless channel are taken into account in the state space of the MDP model. The offloading delay and energy consumption are considered to define the value function of the MDP model, i.e. the objective function. To maximize the value function, Value Iteration Algorithm is used to obtain the optimal offloading policy. According to the policy, tasks of mobile terminals (MTs) are offloaded to the edge cloud or central cloud, or executed locally. The simulation results show that the proposed algorithm can effectively reduce the offloading delay and energy consumption.

scholarly journals Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach

2019 ◽  
Vol 33 ◽  
pp. 7949-7956
Author(s):  
Silviu Pitis
Keyword(s):  
Reinforcement Learning ◽  
Markov Decision Process ◽  
Decision Process ◽  
Value Function ◽  
Discount Factor ◽  
Theoretic Approach ◽  
Decision Theoretic Approach ◽  
Markov Decision ◽  
The Value Function

Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor γ

scholarly journals The Convergence of a Cooperation Markov Decision Process System

Entropy ◽  
2020 ◽  
Vol 22 (9) ◽  
pp. 955
Author(s):  
Xiaoling Mo ◽  
Daoyun Xu ◽  
Zufeng Fu
Keyword(s):  
Markov Decision Process ◽  
Decision Process ◽  
Optimal Strategy ◽  
Value Function ◽  
Single Agent ◽  
Two Agents ◽  
Markov Decision ◽  
Process System ◽  
Multi Agent ◽  
Game Environment

In a general Markov decision progress system, only one agent’s learning evolution is considered. However, considering the learning evolution of a single agent in many problems has some limitations, more and more applications involve multi-agent. There are two types of cooperation, game environment among multi-agent. Therefore, this paper introduces a Cooperation Markov Decision Process (CMDP) system with two agents, which is suitable for the learning evolution of cooperative decision between two agents. It is further found that the value function in the CMDP system also converges in the end, and the convergence value is independent of the choice of the value of the initial value function. This paper presents an algorithm for finding the optimal strategy pair (πk0,πk1) in the CMDP system, whose fundamental task is to find an optimal strategy pair and form an evolutionary system CMDP(πk0,πk1). Finally, an example is given to support the theoretical results.

scholarly journals A Moreau-Yosida regularization for Markov decision processes

2020 ◽  
Vol 40 (1) ◽  
pp. 117-137
Author(s):  
R. Israel Ortega-Gutiérrez ◽  
H. Cruz-Suárez
Keyword(s):  
Markov Decision Process ◽  
Markov Decision Processes ◽  
Optimal Policy ◽  
Decision Process ◽  
Value Function ◽  
Decision Processes ◽  
Original Process ◽  
Optimal Value ◽  
Markov Decision ◽  
Yosida Regularization

This paper addresses a class of sequential optimization problems known as Markov decision processes. These kinds of processes are considered on Euclidean state and action spaces with the total expected discounted cost as the objective function. The main goal of the paper is to provide conditions to guarantee an adequate Moreau-Yosida regularization for Markov decision processes (named the original process). In this way, a new Markov decision process that conforms to the Markov control model of the original process except for the cost function induced via the Moreau-Yosida regularization is established. Compared to the original process, this new discounted Markov decision process has richer properties, such as the differentiability of its optimal value function, strictly convexity of the value function, uniqueness of optimal policy, and the optimal value function and the optimal policy of both processes, are the same. To complement the theory presented, an example is provided.

scholarly journals Markov decision process algorithms for wealth allocation problems with defaultable bonds

2016 ◽  
Vol 48 (2) ◽  
pp. 392-405 ◽  
Author(s):  
Iker Perez ◽  
David Hodge ◽  
Huiling Le
Keyword(s):  
Markov Decision Process ◽  
Decision Process ◽  
Optimization Problem ◽  
Value Function ◽  
Numerical Solutions ◽  
Markov Decision ◽  
Portfolio Optimization Problem ◽  
Wealth Allocation ◽  
Allocation Strategies

Abstract In this paper we are concerned with analysing optimal wealth allocation techniques within a defaultable financial market similar to Bielecki and Jang (2007). We study a portfolio optimization problem combining a continuous-time jump market and a defaultable security; and present numerical solutions through the conversion into a Markov decision process and characterization of its value function as a unique fixed point to a contracting operator. In this paper we analyse allocation strategies under several families of utility functions, and highlight significant portfolio selection differences with previously reported results.

Sign in / Sign up

Export Citation Format

Share Document