Optimal control of random walks, birth and death processes, and queues

1981 ◽  
Vol 13 (01) ◽  
pp. 61-83 ◽  
Author(s):  
Richard Serfozo
Keyword(s):  
Optimal Control ◽  
Random Walks ◽  
Queue Length ◽  
Transition Probabilities ◽  
Transition Rates ◽  
Average Reward ◽  
Birth And Death Processes ◽  
Optimal Policies ◽  
One Step ◽  
Increasing Functions

This is a study of simple random walks, birth and death processes, and M/M/s queues that have transition probabilities and rates that are sequentially controlled at jump times of the processes. Each control action yields a one-step reward depending on the chosen probabilities or transition rates and the state of the process. The aim is to find control policies that maximize the total discounted or average reward. Conditions are given for these processes to have certain natural monotone optimal policies. Under such a policy for the M/M/s queue, for example, the service and arrival rates are non-decreasing and non-increasing functions, respectively, of the queue length. Properties of these policies and a linear program for computing them are also discussed.

Optimal control of random walks, birth and death processes, and queues

1981 ◽  
Vol 13 (1) ◽  
pp. 61-83 ◽  
Author(s):  
Richard Serfozo
Keyword(s):  
Optimal Control ◽  
Random Walks ◽  
Queue Length ◽  
Transition Probabilities ◽  
Transition Rates ◽  
Average Reward ◽  
Birth And Death Processes ◽  
Optimal Policies ◽  
One Step ◽  
Increasing Functions

This is a study of simple random walks, birth and death processes, and M/M/s queues that have transition probabilities and rates that are sequentially controlled at jump times of the processes. Each control action yields a one-step reward depending on the chosen probabilities or transition rates and the state of the process. The aim is to find control policies that maximize the total discounted or average reward. Conditions are given for these processes to have certain natural monotone optimal policies. Under such a policy for the M/M/s queue, for example, the service and arrival rates are non-decreasing and non-increasing functions, respectively, of the queue length. Properties of these policies and a linear program for computing them are also discussed.

scholarly journals Random Walks with Invariant Loop Probabilities: Stereographic Random Walks

Entropy ◽  
2021 ◽  
Vol 23 (6) ◽  
pp. 729
Author(s):  
Miquel Montero
Keyword(s):  
Random Walks ◽  
Stereographic Projection ◽  
Transition Probabilities ◽  
Simple Random Walk ◽  
General Introduction ◽  
Elliptic Case ◽  
One Step ◽  
Underlying Geometry ◽  
Step Transition ◽  
Wide Family

Random walks with invariant loop probabilities comprise a wide family of Markov processes with site-dependent, one-step transition probabilities. The whole family, which includes the simple random walk, emerges from geometric considerations related to the stereographic projection of an underlying geometry into a line. After a general introduction, we focus our attention on the elliptic case: random walks on a circle with built-in reflexing boundaries.

Perturbation theory for unbounded Markov reward processes with applications to queueing

1988 ◽  
Vol 20 (01) ◽  
pp. 99-111 ◽  
Author(s):  
Nico M. Van Dijk
Keyword(s):  
Transition Probabilities ◽  
Arrival Rate ◽  
Average Reward ◽  
Reward Function ◽  
One Step ◽  
Reward Process ◽  
Markov Reward ◽  
Step Transition ◽  
The One ◽  
Queueing Processes

Consider a perturbation in the one-step transition probabilities and rewards of a discrete-time Markov reward process with an unbounded one-step reward function. A perturbation estimate is derived for the finite horizon and average reward function. Results from [3] are hereby extended to the unbounded case. The analysis is illustrated for one- and two-dimensional queueing processes by an M/M/1-queue and an overflow queueing model with an error bound in the arrival rate.

scholarly journals Transient analysis of a queue with queue-length dependent MAP and its application to SS7 network

1999 ◽  
Vol 12 (4) ◽  
pp. 371-392
Author(s):  
Bong Dae Choi ◽  
Sung Ho Choi ◽  
Dan Keun Sung ◽  
Tae-Hee Lee ◽  
Kyu-Seog Song
Keyword(s):  
Congestion Control ◽  
Queue Length ◽  
Transient Analysis ◽  
Transition Probabilities ◽  
Transient Behavior ◽  
Complex Model ◽  
Arrival Process ◽  
One Step ◽  
Step Transition ◽  
The One

We analyze the transient behavior of a Markovian arrival queue with congestion control based on a double of thresholds, where the arrival process is a queue-length dependent Markovian arrival process. We consider Markov chain embedded at arrival epochs and derive the one-step transition probabilities. From these results, we obtain the mean delay and the loss probability of the nth arrival packet. Before we study this complex model, first we give a transient analysis of an MAP/M/1 queueing system without congestion control at arrival epochs. We apply our result to a signaling system No. 7 network with a congestion control based on thresholds.

Perturbation theory for unbounded Markov reward processes with applications to queueing

1988 ◽  
Vol 20 (1) ◽  
pp. 99-111 ◽  
Author(s):  
Nico M. Van Dijk
Keyword(s):  
Transition Probabilities ◽  
Arrival Rate ◽  
Average Reward ◽  
Reward Function ◽  
One Step ◽  
Reward Process ◽  
Markov Reward ◽  
Step Transition ◽  
The One ◽  
Queueing Processes

Consider a perturbation in the one-step transition probabilities and rewards of a discrete-time Markov reward process with an unbounded one-step reward function. A perturbation estimate is derived for the finite horizon and average reward function. Results from [3] are hereby extended to the unbounded case. The analysis is illustrated for one- and two-dimensional queueing processes by an M/M/1-queue and an overflow queueing model with an error bound in the arrival rate.

A review of multistate modelling approaches in monitoring disease progression: Bayesian estimation using the Kolmogorov-Chapman forward equations

2021 ◽  
pp. 096228022199750
Author(s):  
Zvifadzo Matsena Zingoni ◽  
Tobias F Chirwa ◽  
Jim Todd ◽  
Eustasius Musenge
Keyword(s):  
Bayesian Estimation ◽  
Transition Probabilities ◽  
Markov Property ◽  
Likelihood Estimation ◽  
Single Individual ◽  
Transition Rates ◽  
Multistate Models ◽  
Time To Event ◽  
Modeling Framework ◽  
Forward Equations

There are numerous fields of science in which multistate models are used, including biomedical research and health economics. In biomedical studies, these stochastic continuous-time models are used to describe the time-to-event life history of an individual through a flexible framework for longitudinal data. The multistate framework can describe more than one possible time-to-event outcome for a single individual. The standard estimation quantities in multistate models are transition probabilities and transition rates which can be mapped through the Kolmogorov-Chapman forward equations from the Bayesian estimation perspective. Most multistate models assume the Markov property and time homogeneity; however, if these assumptions are violated, an extension to non-Markovian and time-varying transition rates is possible. This manuscript extends reviews in various types of multistate models, assumptions, methods of estimation and data features compatible with fitting multistate models. We highlight the contrast between the frequentist (maximum likelihood estimation) and the Bayesian estimation approaches in the multistate modeling framework and point out where the latter is advantageous. A partially observed and aggregated dataset from the Zimbabwe national ART program was used to illustrate the use of Kolmogorov-Chapman forward equations. The transition rates from a three-stage reversible multistate model based on viral load measurements in WinBUGS were reported.

scholarly journals Current Trends in Random Walks on Random Lattices

Mathematics ◽  
2021 ◽  
Vol 9 (10) ◽  
pp. 1148
Author(s):  
Jewgeni H. Dshalalow ◽  
Ryan T. White
Keyword(s):  
Random Walk ◽  
Random Walks ◽  
Real Time ◽  
Real Space ◽  
Historical Background ◽  
Escape Time ◽  
The Real ◽  
Current Trends ◽  
One Step ◽  
Time Position

In a classical random walk model, a walker moves through a deterministic d-dimensional integer lattice in one step at a time, without drifting in any direction. In a more advanced setting, a walker randomly moves over a randomly configured (non equidistant) lattice jumping a random number of steps. In some further variants, there is a limited access walker’s moves. That is, the walker’s movements are not available in real time. Instead, the observations are limited to some random epochs resulting in a delayed information about the real-time position of the walker, its escape time, and location outside a bounded subset of the real space. In this case we target the virtual first passage (or escape) time. Thus, unlike standard random walk problems, rather than crossing the boundary, we deal with the walker’s escape location arbitrarily distant from the boundary. In this paper, we give a short historical background on random walk, discuss various directions in the development of random walk theory, and survey most of our results obtained in the last 25–30 years, including the very recent ones dated 2020–21. Among different applications of such random walks, we discuss stock markets, stochastic networks, games, and queueing.

scholarly journals Denumerable state continuous time Markov decision processes with unbounded cost and transition rates under average criterion

2002 ◽  
Vol 43 (4) ◽  
pp. 541-557 ◽  
Author(s):  
Xianping Guo ◽  
Weiping Zhu
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Decision Processes ◽  
Transition Rates ◽  
Birth And Death Processes ◽  
Optimality Equation ◽  
Average Criterion ◽  
Markov Decision ◽  
Unbounded Cost ◽  
Queue Model

AbstractIn this paper, we consider denumerable state continuous time Markov decision processes with (possibly unbounded) transition and cost rates under average criterion. We present a set of conditions and prove the existence of both average cost optimal stationary policies and a solution of the average optimality equation under the conditions. The results in this paper are applied to an admission control queue model and controlled birth and death processes.

scholarly journals Stability Analysis for Markovian Jump Neutral Systems with Mixed Delays and Partially Known Transition Rates

2012 ◽  
Vol 2012 ◽  
pp. 1-22 ◽  
Author(s):  
Lianglin Xiong ◽  
Xiaobing Zhou ◽  
Jie Qiu ◽  
Jing Lei
Keyword(s):  
Partial Information ◽  
Transition Probabilities ◽  
Mixed Delays ◽  
Transition Rates ◽  
Neutral Systems ◽  
Markovian Jump ◽  
Markovian Jump System ◽  
Analysis Technique ◽  
Transition Rate Matrix ◽  
Delay Dependent

The delay-dependent stability problem is studied for Markovian jump neutral systems with partial information on transition probabilities, and the considered delays are mixed and model dependent. By constructing the new stochastic Lyapunov-Krasovskii functional, which combined the introduced free matrices with the analysis technique of matrix inequalities, a sufficient condition for the systems with fully known transition rates is firstly established. Then, making full use of the transition rate matrix, the results are obtained for the other case, and the uncertain neutral Markovian jump system with incomplete transition rates is also considered. Finally, to show the validity of the obtained results, three numerical examples are provided.

Sign in / Sign up

Export Citation Format

Share Document