A stochastic control problem with regime switching

This paper studies a stochastic control problem with regime switching in a fairly general abstract setting. Such problems may arise from production planning management. We perform a full mathematical analysis of this stochastic control problem via the HJB equation and verification. The connection of the optimal controls and subgame perfect controls is discussed, and it is shown that the optimal controls solve the generalized HJB equation as well. In a special case we provide a closed form solution.

Partial Diffusion Markov Model of Heterogeneous TCP Link: Optimization with Incomplete Information

The paper presents a new mathematical model of TCP (Transmission Control Protocol) link functioning in a heterogeneous (wired/wireless) channel. It represents a controllable, partially observable stochastic dynamic system. The system state describes the status of the modeled TCP link and expresses it via an unobservable controllable MJP (Markov jump process) with finite-state space. Observations are formed by low-frequency counting processes of packet losses and timeouts and a high-frequency compound Poisson process of packet acknowledgments. The information transmission through the TCP-equipped channel is considered a stochastic control problem with incomplete information. The main idea to solve it is to impose the separation principle on the problem. The paper proposes a mathematical framework and algorithmic support to implement the solution. It includes a solution to the stochastic control problem with complete information, a diffusion approximation of the high-frequency observations, a solution to the MJP state filtering problem given the observations with multiplicative noises, and a numerical scheme of the filtering algorithm. The paper also contains the results of a comparative study of the proposed state-based congestion control algorithm with the contemporary TCP versions: Illinois, CUBIC, Compound, and BBR (Bottleneck Bandwidth and RTT).