scholarly journals On the total reward variance for continuous-time Markov reward chains

2006 ◽  
Vol 43 (04) ◽  
pp. 1044-1052 ◽  
Author(s):  
Nico M. Van Dijk ◽  
Karel Sladký
Keyword(s):  
Growth Rate ◽  
Discrete Time ◽  
Continuous Time ◽  
Asymptotically Linear ◽  
State Spaces ◽  
Total Reward ◽  
Finite State ◽  
Markov Reward ◽  
Discrete Time Case

As an extension of the discrete-time case, this note investigates the variance of the total cumulative reward for continuous-time Markov reward chains with finite state spaces. The results correspond to discrete-time results. In particular, the variance growth rate is shown to be asymptotically linear in time. Expressions are provided to compute this growth rate.

scholarly journals On the total reward variance for continuous-time Markov reward chains

2006 ◽  
Vol 43 (4) ◽  
pp. 1044-1052 ◽  
Author(s):  
Nico M. Van Dijk ◽  
Karel Sladký
Keyword(s):  
Growth Rate ◽  
Discrete Time ◽  
Continuous Time ◽  
Asymptotically Linear ◽  
State Spaces ◽  
Total Reward ◽  
Finite State ◽  
Markov Reward ◽  
Discrete Time Case

As an extension of the discrete-time case, this note investigates the variance of the total cumulative reward for continuous-time Markov reward chains with finite state spaces. The results correspond to discrete-time results. In particular, the variance growth rate is shown to be asymptotically linear in time. Expressions are provided to compute this growth rate.

On quasi-stationary distributions in absorbing continuous-time finite Markov chains

1967 ◽  
Vol 4 (1) ◽  
pp. 192-196 ◽  
Author(s):  
J. N. Darroch ◽  
E. Seneta
Keyword(s):  
Markov Chains ◽  
Discrete Time ◽  
Continuous Time ◽  
Present Note ◽  
Time Parameter ◽  
Stationary Distributions ◽  
Finite State ◽  
Absorbing Markov Chains ◽  
Finite Markov Chains ◽  
Finite State Space

In a recent paper, the authors have discussed the concept of quasi-stationary distributions for absorbing Markov chains having a finite state space, with the further restriction of discrete time. The purpose of the present note is to summarize the analogous results when the time parameter is continuous.

The Computation of Average Optimal Policies in Denumerable State Markov Decision Chains

1997 ◽  
Vol 29 (01) ◽  
pp. 114-137
Author(s):  
Linn I. Sennott
Keyword(s):  
Discrete Time ◽  
Average Cost ◽  
Queueing Systems ◽  
State Spaces ◽  
Original Process ◽  
Optimal Policies ◽  
Finite State ◽  
Markov Decision ◽  
Optimal Average ◽  
Infinite State

This paper studies the expected average cost control problem for discrete-time Markov decision processes with denumerably infinite state spaces. A sequence of finite state space truncations is defined such that the average costs and average optimal policies in the sequence converge to the optimal average cost and an optimal policy in the original process. The theory is illustrated with several examples from the control of discrete-time queueing systems. Numerical results are discussed.

On a Conjecture of Bollobás and Brightwell Concerning Random Walks on Product Graphs

1998 ◽  
Vol 7 (4) ◽  
pp. 397-401 ◽  
Author(s):  
OLLE HÄGGSTRÖM
Keyword(s):  
Random Walk ◽  
Random Walks ◽  
Discrete Time ◽  
Complete Graph ◽  
Continuous Time ◽  
Product Graph ◽  
Product Graphs ◽  
Continuous Time Random Walks ◽  
Discrete Time Case

We consider continuous time random walks on a product graph G×H, where G is arbitrary and H consists of two vertices x and y linked by an edge. For any t>0 and any a, b∈V(G), we show that the random walk starting at (a, x) is more likely to have hit (b, x) than (b, y) by time t. This contrasts with the discrete time case and proves a conjecture of Bollobás and Brightwell. We also generalize the result to cases where H is either a complete graph on n vertices or a cycle on n vertices.

scholarly journals A Model of Indel Evolution by Finite-State, Continuous-Time Machines

Genetics ◽  
2020 ◽  
Vol 216 (4) ◽  
pp. 1187-1204
Author(s):  
Ian Holmes
Keyword(s):  
Differential Equations ◽  
Continuous Time ◽  
Markov Models ◽  
Transition Probabilities ◽  
Simulated Data ◽  
Coarse Graining ◽  
Automata Theory ◽  
State Spaces ◽  
Finite State ◽  
Related Approach

We introduce a systematic method of approximating finite-time transition probabilities for continuous-time insertion-deletion models on sequences. The method uses automata theory to describe the action of an infinitesimal evolutionary generator on a probability distribution over alignments, where both the generator and the alignment distribution can be represented by pair hidden Markov models (HMMs). In general, combining HMMs in this way induces a multiplication of their state spaces; to control this, we introduce a coarse-graining operation to keep the state space at a constant size. This leads naturally to ordinary differential equations for the evolution of the transition probabilities of the approximating pair HMM. The TKF91 model emerges as an exact solution to these equations for the special case of single-residue indels. For the more general case of multiple-residue indels, the equations can be solved by numerical integration. Using simulated data, we show that the resulting distribution over alignments, when compared to previous approximations, is a better fit over a broader range of parameters. We also propose a related approach to develop differential equations for sufficient statistics to estimate the underlying instantaneous indel rates by expectation maximization. Our code and data are available at https://github.com/ihh/trajectory-likelihood.

MONOTONICITY AND CONVEXITY OF SOME FUNCTIONS ASSOCIATED WITH DENUMERABLE MARKOV CHAINS AND THEIR APPLICATIONS TO QUEUING SYSTEMS

2005 ◽  
Vol 20 (1) ◽  
pp. 67-86 ◽  
Author(s):  
Hai-Bo Yu ◽  
Qi-Ming He ◽  
Hanqin Zhang
Keyword(s):  
Markov Chains ◽  
Discrete Time ◽  
Stochastic Dominance ◽  
Continuous Time ◽  
Queue Length ◽  
Queuing Theory ◽  
Queuing Systems ◽  
Discrete Time Case ◽  
Monotone Matrix

Motivated by various applications in queuing theory, this article is devoted to the monotonicity and convexity of some functions associated with discrete-time or continuous-time denumerable Markov chains. For the discrete-time case, conditions for the monotonicity and convexity of the functions are obtained by using the properties of stochastic dominance and monotone matrix. For the continuous-time case, by using the uniformization technique, similar results are obtained. As an application, the results are applied to analyze the monotonicity and convexity of functions associated with the queue length of some queuing systems.

NONPARAMETRIC NONSTATIONARITY TESTS

2013 ◽  
Vol 30 (1) ◽  
pp. 127-149 ◽  
Author(s):  
Federico M. Bandi ◽  
Valentina Corradi
Keyword(s):  
Time Series ◽  
Discrete Time ◽  
Continuous Time ◽  
Diffusion Processes ◽  
Nonlinear Process ◽  
Additive Functional ◽  
Finite Sample ◽  
Occupation Times ◽  
Discrete Time Case ◽  
Continuous Time Processes

We propose additive functional-based nonstationarity tests that exploit the different divergence rates of the occupation times of a (possibly nonlinear) process under the null of nonstationarity (stationarity) versus the alternative of stationarity (nonstationarity). We consider both discrete-time series and continuous-time processes. The discrete-time case covers Harris recurrent Markov chains and integrated processes. The continuous-time case focuses on Harris recurrent diffusion processes. Notwithstanding finite-sample adjustments discussed in the paper, the proposed tests are simple to implement and rely on tabulated critical values. Simulations show that their size and power properties are satisfactory. Our robustness to nonlinear dynamics provides a solution to the typical inconsistency problem between assumed linearity of a time series for the purpose of nonstationarity testing and subsequent nonlinear inference.

Approximations of general discrete time queues by discrete time queues with arrivals modulated by finite chains

1997 ◽  
Vol 29 (04) ◽  
pp. 1039-1059
Author(s):  
Vinod Sharma
Keyword(s):  
Markov Chains ◽  
Discrete Time ◽  
Continuous Time ◽  
Point Processes ◽  
Marked Point ◽  
Marked Point Processes ◽  
Marked Point Process ◽  
Discrete Time Queues ◽  
Time Queues ◽  
Finite State

Recently, Asmussen and Koole (Journal of Applied Probability 30, pp. 365–372) showed that any discrete or continuous time marked point process can be approximated by a sequence of arrival streams modulated by finite state continuous time Markov chains. If the original process is customer (time) stationary then so are the approximating processes. Also, the moments in the stationary case converge. For discrete marked point processes we construct a sequence of discrete processes modulated by discrete time finite state Markov chains. All the above features of approximating sequences of Asmussen and Koole continue to hold. For discrete arrival sequences (to a queue) which are modulated by a countable state Markov chain we form a different sequence of approximating arrival streams by which, unlike in the Asmussen and Koole case, even the stationary moments of waiting times can be approximated. Explicit constructions for the output process of a queue and the total input process of a discrete time Jackson network with these characteristics are obtained.

Sign in / Sign up

Export Citation Format

Share Document