ERROR BOUNDS FOR OCCUPATION MEASURE OF SINGULARLY PERTURBED MARKOV CHAINS INCLUDING TRANSIENT STATES

2000 ◽  
Vol 14 (4) ◽  
pp. 511-531 ◽  
Author(s):  
G. Yin ◽  
Q. Zhang ◽  
Q. G. Liu
Keyword(s):  
Markov Chains ◽  
Error Bounds ◽  
Queueing Networks ◽  
Large Scale ◽  
Singularly Perturbed ◽  
Planning System ◽  
Nonstationary Processes ◽  
Transient States ◽  
Asymptotic Error ◽  
Occupation Measures

Motivated by many applications in production planning, system reliability, queueing networks, and wireless communication, this work is devoted to singularly perturbed Markov chains with finite states. Focusing on nonstationary processes with the inclusion of transient states, asymptotic error bounds of a sequence of suitably scaled occupation measures are derived. The main tools used include martingales and differential equations. The results are useful for analyzing structural properties of the underlying Markov chains and for designing nearly optimal and hierarchical controls of large-scale and complex systems.

Discrete-time singularly perturbed Markov chains: aggregation, occupation measures, and switching diffusion limit

2003 ◽  
Vol 35 (2) ◽  
pp. 449-476 ◽  
Author(s):  
G. Yin ◽  
Q. Zhang ◽  
G. Badowski
Keyword(s):  
Markov Chains ◽  
Discrete Time ◽  
Continuous Time ◽  
Communication Systems ◽  
Large Scale ◽  
Optimization Problems ◽  
Singularly Perturbed ◽  
Diffusion Limit ◽  
Switching Diffusion ◽  
Occupation Measures

This work is devoted to asymptotic properties of singularly perturbed Markov chains in discrete time. The motivation stems from applications in discrete-time control and optimization problems, manufacturing and production planning, stochastic networks, and communication systems, in which finite-state Markov chains are used to model large-scale and complex systems. To reduce the complexity of the underlying system, the states in each recurrent class are aggregated into a single state. Although the aggregated process may not be Markovian, its continuous-time interpolation converges to a continuous-time Markov chain whose generator is a function determined by the invariant measures of the recurrent states. Sequences of occupation measures are defined. A mean square estimate on a sequence of unscaled occupation measures is obtained. Furthermore, it is proved that a suitably scaled sequence of occupation measures converges to a switching diffusion.

Discrete-time singularly perturbed Markov chains: aggregation, occupation measures, and switching diffusion limit

2003 ◽  
Vol 35 (02) ◽  
pp. 449-476 ◽  
Author(s):  
G. Yin ◽  
Q. Zhang ◽  
G. Badowski
Keyword(s):  
Markov Chains ◽  
Discrete Time ◽  
Continuous Time ◽  
Communication Systems ◽  
Large Scale ◽  
Optimization Problems ◽  
Singularly Perturbed ◽  
Diffusion Limit ◽  
Switching Diffusion ◽  
Occupation Measures

This work is devoted to asymptotic properties of singularly perturbed Markov chains in discrete time. The motivation stems from applications in discrete-time control and optimization problems, manufacturing and production planning, stochastic networks, and communication systems, in which finite-state Markov chains are used to model large-scale and complex systems. To reduce the complexity of the underlying system, the states in each recurrent class are aggregated into a single state. Although the aggregated process may not be Markovian, its continuous-time interpolation converges to a continuous-time Markov chain whose generator is a function determined by the invariant measures of the recurrent states. Sequences of occupation measures are defined. A mean square estimate on a sequence of unscaled occupation measures is obtained. Furthermore, it is proved that a suitably scaled sequence of occupation measures converges to a switching diffusion.

scholarly journals Control of singularly perturbed Markov chains: A numerical study

2003 ◽  
Vol 45 (1) ◽  
pp. 49-74 ◽  
Author(s):  
H. Yang ◽  
G. Yin ◽  
K. Yin ◽  
Q. Zhang
Keyword(s):  
Markov Chains ◽  
Discrete Time ◽  
Large Scale ◽  
Numerical Study ◽  
Asymptotic Properties ◽  
Linear Quadratic Regulator ◽  
Singularly Perturbed ◽  
Optimal Controls ◽  
Linear Quadratic ◽  
Markov Decision

AbstractThis work is devoted to numerical studies of nearly optimal controls of systems driven by singularly perturbed Markov chains. Our approach is based on the ideas of hierarchical controls applicable to many large-scale systems. A discrete-time linear quadratic control problem is examined. Its corresponding limit system is derived. The associated asymptotic properties and near optimality are demonstrated by numerical examples. Numerical experiments for a continuous-time hybrid linear quadratic regulator with Gaussian disturbances and a discrete-time Markov decision process are also presented. The numerical results have not only supported our theoretical findings but also provided insights for further applications.

Dynamic Control of Logistics Queueing Networks for Large-Scale Fleet Management

1998 ◽  
Vol 32 (2) ◽  
pp. 90-109 ◽  
Author(s):  
Warren B. Powell ◽  
Tassio A. Carvalho
Keyword(s):  
Queueing Networks ◽  
Large Scale ◽  
Dynamic Control ◽  
Fleet Management
Sign in / Sign up

Export Citation Format

Share Document