What is the state space of the world for real animals?

A key concept in reinforcement learning (RL) is that of a state space. A state space is an abstract representation of the world using which statistical relations in the world can be described. The simplest form of RL, model free RL, is widely applied to explain animal behavior in numerous neuroscientific studies. More complex RL versions assume that animals build and store an explicit model of the world in memory. To apply these approaches to explain animal behavior, typical neuroscientific RL models make assumptions about the underlying state space formed by animals, especially regarding the representation of time. Here, we explicitly list these assumptions and show that they have several problematic implications. We propose a solution for these problems by using a continuous time Markov renewal process model of the state space. We hope that our explicit treatment results in a serious consideration of these issues when applying RL models to real animals.

Non-Homogeneous Semi-Markov and Markov Renewal Processes and Change of Measure in Credit Risk

For a G-inhomogeneous semi-Markov chain and G-inhomogeneous Markov renewal processes, we study the change from real probability measure into a forward probability measure. We find the values of risky bonds using the forward probabilities that the bond will not default up to maturity time for both processes. It is established in the form of a theorem that the forward probability measure does not alter the semi Markov structure. In addition, foundation of a G-inhohomogeneous Markov renewal process is done and a theorem is provided where it is proved that the Markov renewal process is maintained under the forward probability measure. We show that for an inhomogeneous semi-Markov there are martingales that characterize it. We show that the same is true for a Markov renewal processes. We discuss in depth the calibration of the G-inhomogeneous semi-Markov chain model and propose an algorithm for it. We conclude with an application for risky bonds.

Analysis of the Message Propagation Speed in VANET with Disconnected RSUs

Vehicular ad-hoc networks (VANETs), which are networks of communicating vehicles, provide the essential infrastructure for intelligent transportation systems. Thanks to the significant research efforts to develop the technological background of VANETs, intelligent transportation systems are nowadays becoming a reality. The emergence of VANETs has triggered a lot of research aimed at developing mathematical models in order to gain insight into the dynamics of the communication and to support network planning. In this paper we consider the message propagation speed on the highway, where messages can be exchanged not only between the vehicles, but also between the road-side infrastructure and the vehicles as well. In our scenario, alert messages are generated by a static message source constantly. Relying on an appropriately defined Markov renewal process, we characterize the message passing process between the road-side units, derive the speed of the message propagation, and provide the transient distribution of the distance where the message is available. Our results make it possible to determine the optimal distance between road-side units (RSUs) and to calculate the effect of speed restrictions on message propagation.

A Markov Renewal Chain Model for Forecasting Earthquakes in Bangladesh Region

In this paper we have proposed a Weibull Markov renewal process to model earthquakes occurred in and around Bangladesh from 1961 to 2013. The process assumes that the sequence of earthquakes is a Markov chain and the sojourn time distribution is a Weibull random variable that depends only on two successive earthquakes. We estimated the parameters of the models along with transition probabilities using maximum likelihood method. The transient behavior of earthquake occurrences was investigated in details and probability forecasts were calculated for different lengths of time interval using the fitted model. We also investigated the stationary behavior of earthquake occurrences in Bangladesh region. Dhaka Univ. J. Sci. 65(1): 15-20, 2017 (January)

Large deviations for the empirical measure of heavy-tailed Markov renewal processes

A large deviations principle is established for the joint law of the empirical measure and the flow measure of a Markov renewal process on a finite graph. We do not assume any bound on the arrival times, allowing heavy-tailed distributions. In particular, the rate function is in general degenerate (it has a nontrivial set of zeros) and not strictly convex. These features show a behaviour highly different from what one may guess with a heuristic Donsker‒Varadhan analysis of the problem.

Accurate calculations of Stationary Distributions and Mean First Passage Times in Markov Renewal Processes and Markov Chains

This article describes an accurate procedure for computing the mean first passage times of a finite irreducible Markov chain and a Markov renewal process. The method is a refinement to the Kohlas, Zeit fur Oper Res, 30, 197–207, (1986) procedure. The technique is numerically stable in that it doesn’t involve subtractions. Algebraic expressions for the special cases of one, two, three and four states are derived.Aconsequence of the procedure is that the stationary distribution of the embedded Markov chain does not need to be derived in advance but can be found accurately from the derived mean first passage times. MatLab is utilized to carry out the computations, using some test problems from the literature.