Discrete-time Markov chains with two-time scales and a countable state space: limit results and queueing applications

In this paper, we give the definition of tree-indexed Markov chains in random environment with countable state space, and then study the realization of Markov chain indexed by a tree in random environment. Finally, we prove the strong law of large numbers and Shannon–McMillan theorem for Markov chains indexed by a Cayley tree in a Markovian environment with countable state space.

Download Full-text

Criteria for the non-ergodicity of stochastic processes: application to the exponential back-off protocol

Journal of Applied Probability ◽

10.1017/s0021900200030990 ◽

1987 ◽

Vol 24 (02) ◽

pp. 347-354 ◽

Cited By ~ 1

Author(s):

Guy Fayolle ◽

Rudolph Iasnogorodski

Keyword(s):

Stochastic Process ◽

Markov Chain ◽

Stochastic Processes ◽

State Space ◽

Discrete Time ◽

Random Variables ◽

Countable State Space ◽

Countable State ◽

New Criteria

In this paper, we present some simple new criteria for the non-ergodicity of a stochastic process (Yn ), n ≧ 0 in discrete time, when either the upward or downward jumps are majorized by i.i.d. random variables. This situation is encountered in many practical situations, where the (Yn ) are functionals of some Markov chain with countable state space. An application to the exponential back-off protocol is described.

Download Full-text

Two Approaches to the Construction of Perturbation Bounds for Continuous-Time Markov Chains

Mathematics ◽

10.3390/math8020253 ◽

2020 ◽

Vol 8 (2) ◽

pp. 253 ◽

Cited By ~ 1

Author(s):

Alexander Zeifman ◽

Victor Korolev ◽

Yacov Satin

Keyword(s):

Markov Chains ◽

State Space ◽

Continuous Time ◽

Countable State Space ◽

Perturbation Bounds ◽

Queuing Models ◽

Continuous Time Markov Chains ◽

Quantitative Estimates ◽

Countable State

This paper is largely a review. It considers two main methods used to study stability and to obtain appropriate quantitative estimates of perturbations of (inhomogeneous) Markov chains with continuous time and a finite or countable state space. An approach is described to the construction of perturbation estimates for the main five classes of such chains associated with queuing models. Several specific models are considered for which the limit characteristics and perturbation bounds for admissible “perturbed” processes are calculated.

Download Full-text

Constrained Discounted Markov Decision Chains

Probability in the Engineering and Informational Sciences ◽

10.1017/s0269964800002230 ◽

1991 ◽

Vol 5 (4) ◽

pp. 463-475 ◽

Cited By ~ 26

Author(s):

Linn I. Sennott

Keyword(s):

State Space ◽

Discrete Time ◽

Queueing Systems ◽

Operating Cost ◽

Countable State Space ◽

Countable State ◽

Markov Decision ◽

Holding Cost

A Markov decision chain with countable state space incurs two types of costs: an operating cost and a holding cost. The objective is to minimize the expected discounted operating cost, subject to a constraint on the expected discounted holding cost. The existence of an optimal randomized simple policy is proved. This is a policy that randomizes between two stationary policies, that differ in at most one state. Several examples from the control of discrete time queueing systems are discussed.

Download Full-text

Quasi-ergodicity for non-homogeneous continuous-time Markov chains

Journal of Applied Probability ◽

10.2307/3214422 ◽

1989 ◽

Vol 26 (3) ◽

pp. 643-648 ◽

Cited By ~ 7

Author(s):

A. I. Zeifman

Keyword(s):

Differential Equation ◽

Markov Chain ◽

Markov Chains ◽

State Space ◽

Continuous Time ◽

Sufficient Conditions ◽

Continuous Time Markov Chain ◽

Countable State Space ◽

Continuous Time Markov Chains ◽

Countable State

We consider a non-homogeneous continuous-time Markov chain X(t) with countable state space. Definitions of uniform and strong quasi-ergodicity are introduced. The forward Kolmogorov system for X(t) is considered as a differential equation in the space of sequences l1. Sufficient conditions for uniform quasi-ergodicity are deduced from this equation. We consider conditions of uniform and strong ergodicity in the case of proportional intensities.

Download Full-text

Integral representations of transition probabilities and serial covariances of certain markov chains

Journal of Applied Probability ◽

10.1017/s0021900200026681 ◽

1969 ◽

Vol 6 (03) ◽

pp. 648-659 ◽

Cited By ~ 3

Author(s):

D. J. Daley

Keyword(s):

Markov Chains ◽

State Space ◽

Spectral Measure ◽

Transition Probabilities ◽

Integral Representations ◽

Stationary Processes ◽

Birth And Death Processes ◽

Countable State Space ◽

Completely Monotonic ◽

Countable State

This note is concerned with integral representations of some transition probabilities of countable state space Markov chains embedded in birth and death processes, of the serial covariances of functions defined on such chains when stationary, and finally the properties of the spectral measure of stationary processes with monotonically decreasing or completely monotonic serial covariances.

Download Full-text

Criteria for the non-ergodicity of stochastic processes: application to the exponential back-off protocol

Journal of Applied Probability ◽

10.2307/3214259 ◽

1987 ◽

Vol 24 (2) ◽

pp. 347-354 ◽

Cited By ~ 4

Author(s):

Guy Fayolle ◽

Rudolph Iasnogorodski

Keyword(s):

Stochastic Process ◽

Markov Chain ◽

Stochastic Processes ◽

State Space ◽

Discrete Time ◽

Random Variables ◽

Countable State Space ◽

Countable State ◽

New Criteria

In this paper, we present some simple new criteria for the non-ergodicity of a stochastic process (Yn), n ≧ 0 in discrete time, when either the upward or downward jumps are majorized by i.i.d. random variables. This situation is encountered in many practical situations, where the (Yn) are functionals of some Markov chain with countable state space. An application to the exponential back-off protocol is described.

Download Full-text