Equilibrium in misspecified Markov decision processes

We provide an equilibrium framework for modeling the behavior of an agent who holds a simplified view of a dynamic optimization problem. The agent faces a Markov decision process, where a transition probability function determines the evolution of a state variable as a function of the previous state and the agent's action. The agent is uncertain about the true transition function and has a prior over a set of possible transition functions; this set reflects the agent's (possibly simplified) view of her environment and may not contain the true function. We define an equilibrium concept and provide conditions under which it characterizes steady‐state behavior when the agent updates her beliefs using Bayes' rule.

The Strassen invariance principle for certain non-stationary Markov–Feller chains

We propose certain conditions implying the functional law of the iterated logarithm (the Strassen invariance principle) for some general class of non-stationary Markov–Feller chains. This class may be briefly specified by the following two properties: firstly, the transition operator of the chain under consideration enjoys a non-linear Lyapunov-type condition, and secondly, there exists an appropriate Markovian coupling whose transition probability function can be decomposed into two parts, one of which is contractive and dominant in some sense. Our criterion may serve as a useful tool in verifying the functional law of the iterated logarithm for certain random dynamical systems, developed e.g. in biology and population dynamics. In the final part of the paper we present an example application of our main theorem to a mathematical model describing stochastic dynamics of gene expression.

A method for the calculation of characteristics for the solution to stochastic differential equations

In this work, a new numerical method to calculate the characteristics of the solution to stochastic differential equations is presented. This method is based on the Fokker–Planck equation for the transition probability function and the representation of the transition probability function by means of eigenfunctions of the Fokker–Planck operator. The results of the numerical experiments are presented.

On the metrical theory of a non-regular continued fraction expansion

We introduced a new continued fraction expansions in our previous paper. For these expansions, we show the Brodén-Borel-Lévy type formula. Furthermore, we compute the transition probability function from this and the symbolic dynamical system of the natural number with the unilateral shift.

A Better ACA for Routing and Wavelength Assignment in ASON with Supporting Different SLA of Services and 10Pbps Network Capacity

Dynamic RWA algorithm which supports differentiating business service quality levels becomes a hot research in optical networks. This paper proposes a better ant colony algorithm for routing and wavelength assignment in ASON with supporting differentiating SLA of services and 10P bps network capacity. Firstly, the algorithm adds the concept of business interval period M to dynamically adjust the wavelength grouped in different periods So as to better meet the sercice requests of different levels; Secondly the algorithm adds a link selection control factor in the transition probability function ACA in order to achieve a better link load balancing. Simulation results show that after joining the cyclical and link selection control factor, blocking rate becomes lower and the load of network becomes more evenly.

Markov Chain Models for the Stochastic Modeling of Pitting Corrosion

The stochastic nature of pitting corrosion of metallic structures has been widely recognized. It is assumed that this kind of deterioration retains no memory of the past, so only the current state of the damage influences its future development. This characteristic allows pitting corrosion to be categorized as a Markov process. In this paper, two different models of pitting corrosion, developed using Markov chains, are presented. Firstly, a continuous-time, nonhomogeneous linear growth (pure birth) Markov process is used to model external pitting corrosion in underground pipelines. A closed-form solution of the system of Kolmogorov's forward equations is used to describe the transition probability function in a discrete pit depth space. The transition probability function is identified by correlating the stochastic pit depth mean with the empirical deterministic mean. In the second model, the distribution of maximum pit depths in a pitting experiment is successfully modeled after the combination of two stochastic processes: pit initiation and pit growth. Pit generation is modeled as a nonhomogeneous Poisson process, in which induction time is simulated as the realization of a Weibull process. Pit growth is simulated using a nonhomogeneous Markov process. An analytical solution of Kolmogorov's system of equations is also found for the transition probabilities from the first Markov state. Extreme value statistics is employed to find the distribution of maximum pit depths.

Markov Chain Model Helps Predict Pitting Corrosion Depth and Rate in Underground Pipelines

A continuous-time, non-homogenous pure birth Markov chain serves to model external pitting corrosion in buried pipelines. The analytical solution of Kolmogorov’s forward equations for this type of Markov process gives the transition probability function in a discrete space of pit depths. The transition probability function can be completely identified by making a correlation between the stochastic pit depth mean and the deterministic mean obtained experimentally. Previously reported Monte Carlo simulations have been used for the prediction of the evolution of the pit depth distribution mean value with time for different soil types. The simulated pit depth distributions are used to develop a stochastic model based on Markov chains to predict the progression of pitting corrosion depth and rate distributions from the observed soil properties and pipeline coating characteristics. The proposed model can also be applied to pitting corrosion data from repeated in-line pipeline inspections. Real-life case studies presented in this work show how pipeline inspection and maintenance planning can be improved through the use of the proposed Markovian model for pitting corrosion.

Computational and estimation procedures in multidimensional right-shift processes and some applications

A right-shift process is a Markov process with multidimensional finite state space on which the infinitesimal transition movement is a shifting of one unit from one coordinate to some other to its right. A multidimensional right-shift process consists of v ≧ 1 concurrent and dependent right-shift processes. In this paper applications of multidimensional right-shift processes to some well-known examples from epidemic theory, queueing theory and the Beetle probblem due to Lucien LeCam are discussed. A transformation which orders the Kolmogorov forward equations into a triangular array is provided and some computational procedures for solving the resulting system of equations are presented. One of these procedures is concerned with the problem of evaluating a given transition probability function rather than obtaining the solution to the complete system of forward equations. This particular procedure is applied to the problem of estimating the parameters of a multidimensional right-shift process which is observed at only a finite number of fixed timepoints.