scholarly journals Optimizing Age Penalty in Time-Varying Networks with Markovian and Error-Prone Channel State

Entropy ◽  
2021 ◽  
Vol 23 (1) ◽  
pp. 91
Author(s):  
Yuchao Chen ◽  
Haoyue Tang ◽  
Jintao Wang ◽  
Jian Song
Keyword(s):  
Optimal Policy ◽  
Infinite Horizon ◽  
Base Station ◽  
Optimal Decision ◽  
Time Varying ◽  
Sensitive Data ◽  
Multiple Sensors ◽  
Data Freshness ◽  
Markov Decision ◽  
Time Varying Networks

In this paper, we consider a scenario where the base station (BS) collects time-sensitive data from multiple sensors through time-varying and error-prone channels. We characterize the data freshness at the terminal end through a class of monotone increasing functions related to Age of information (AoI). Our goal is to design an optimal policy to minimize the average age penalty of all sensors in infinite horizon under bandwidth and power constraint. By formulating the scheduling problem into a constrained Markov decision process (CMDP), we reveal the threshold structure for the optimal policy and approximate the optimal decision by solving a truncated linear programming (LP). Finally, a bandwidth-truncated policy is proposed to satisfy both power and bandwidth constraint. Through theoretical analysis and numerical simulations, we prove the proposed policy is asymptotic optimal in the large sensor regime.

Markov Decision Models for Optimal Decision Making in Bottlenecks Identification and Mitigation

2015 ◽  
Author(s):  
Yang Li ◽  
Qing Chang ◽  
Xiaoning Jin ◽  
Jun Ni
Keyword(s):  
Optimal Policy ◽  
Data Driven ◽  
Optimal Decision ◽  
Stochastic Methods ◽  
Bottleneck Analysis ◽  
Markov Decision Models ◽  
Markov Decision ◽  
Optimal Decision Making

Existing methods for bottleneck detection can be categorized into two: methods based on stochastic analysis and methods based on data-driven analysis. The stochastic methods are accurate in estimating bottlenecks in long term, ignoring the current improvement opportunities, while the data-driven methods tend to do the opposite. In this paper, we develop an optimal policy to integrate the two methods based on Markov decision theory. The characterization of the optimal policy is provided. In addition, to implement the policy, the optimal frequency for carrying out bottleneck analysis is investigated. Numerical experiment is performed to validate the effectiveness of the optimal policy and compare it to the existing methods.

scholarly journals Scheduling to Minimize Age of Incorrect Information with Imperfect Channel State Information

Entropy ◽  
2021 ◽  
Vol 23 (12) ◽  
pp. 1572
Author(s):  
Yutao Chen ◽  
Anthony Ephremides
Keyword(s):  
Structural Properties ◽  
Channel State Information ◽  
Optimal Policy ◽  
Base Station ◽  
Imperfect Channel State Information ◽  
Index Policy ◽  
Channel State ◽  
State Information ◽  
Markov Decision ◽  
Incorrect Information

In this paper, we study a slotted-time system where a base station needs to update multiple users at the same time. Due to the limited resources, only part of the users can be updated in each time slot. We consider the problem of minimizing the Age of Incorrect Information (AoII) when imperfect Channel State Information (CSI) is available. Leveraging the notion of the Markov Decision Process (MDP), we obtain the structural properties of the optimal policy. By introducing a relaxed version of the original problem, we develop the Whittle’s index policy under a simple condition. However, indexability is required to ensure the existence of Whittle’s index. To avoid indexability, we develop Indexed priority policy based on the optimal policy for the relaxed problem. Finally, numerical results are laid out to showcase the application of the derived structural properties and highlight the performance of the developed scheduling policies.

scholarly journals A Version of the Euler Equation in Discounted Markov Decision Processes

2012 ◽  
Vol 2012 ◽  
pp. 1-16 ◽  
Author(s):  
H. Cruz-Suárez ◽  
G. Zacarías-Espinoza ◽  
V. Vázquez-Guevara
Keyword(s):  
Control Problem ◽  
Euler Equation ◽  
Markov Decision Processes ◽  
Optimal Policy ◽  
Infinite Horizon ◽  
Decision Processes ◽  
Value Iteration ◽  
Programming Technique ◽  
Iteration Functions ◽  
Markov Decision

This paper deals with Markov decision processes (MDPs) on Euclidean spaces with an infinite horizon. An approach to study this kind of MDPs is using the dynamic programming technique (DP). Then the optimal value function is characterized through the value iteration functions. The paper provides conditions that guarantee the convergence of maximizers of the value iteration functions to the optimal policy. Then, using the Euler equation and an envelope formula, the optimal solution of the optimal control problem is obtained. Finally, this theory is applied to a linear-quadratic control problem in order to find its optimal policy.

OPTIMALITY OF FOUR-THRESHOLD POLICIES IN INVENTORY SYSTEMS WITH CUSTOMER RETURNS AND BORROWING/STORAGE OPTIONS

2005 ◽  
Vol 19 (1) ◽  
pp. 45-71 ◽  
Author(s):  
Eugene A. Feinberg ◽  
Mark E. Lewis
Keyword(s):  
Optimal Policy ◽  
Infinite Horizon ◽  
Sufficient Conditions ◽  
Inventory Systems ◽  
Linear Ordering ◽  
Optimal Decision ◽  
Cash Management ◽  
Threshold Policies ◽  
Customer Returns ◽  
And Storage

Consider a single-commodity inventory system in which the demand is modeled by a sequence of independent and identically distributed random variables that can take negative values. Such problems have been studied in the literature under the namecash managementand relate to the variations of the on-hand cash balances of financial institutions. The possibility of a negative demand also models product returns in inventory systems. This article studies a model in which, in addition to standard ordering and scrapping decisions seen in the cash management models, the decision-maker can borrow and store some inventory for one period of time. For problems with back orders, zero setup costs, and linear ordering, scrapping, borrowing, and storage costs, we show that an optimal policy has a simple four-threshold structure. These thresholds, in a nondecreasing order, are order-up-to, borrow-up-to, store-down-to, and scrap-down-to levels; that is, if the inventory position is too low, an optimal policy is to order up to a certain level and then borrow up to a higher level. Analogously, if the inventory position is too high, the optimal decision is to reduce the inventory to a certain point, after which one should store some of the inventory down to a lower threshold. This structure holds for the finite and infinite horizon discounted expected cost criteria and for the average cost per unit time criterion. We also provide sufficient conditions when the borrowing and storage options should not be used. In order to prove our results for average costs per unit time, we establish sufficient conditions when the optimality equations hold for a Markov decision process with an uncountable state space, noncompact action sets, and unbounded costs.

scholarly journals Impact of information diffusion on epidemic spreading in partially mapping two-layered time-varying networks

2021 ◽  
Vol 105 (4) ◽  
pp. 3819-3833
Author(s):  
Haili Guo ◽  
Qian Yin ◽  
Chengyi Xia ◽  
Matthias Dehmer
Keyword(s):  
Information Diffusion ◽  
Time Varying ◽  
Epidemic Spreading ◽  
Time Varying Networks

scholarly journals Epidemic Processes Over Time-Varying Networks

2018 ◽  
Vol 5 (3) ◽  
pp. 1322-1334 ◽  
Author(s):  
Philip E. Pare ◽  
Carolyn L. Beck ◽  
Angelia Nedic
Keyword(s):  
Time Varying ◽  
Time Varying Networks ◽  
Over Time

scholarly journals Dynamic Task Migration Combining Energy Efficiency and Load Balancing Optimization in Three-Tier UAV-Enabled Mobile Edge Computing System

Electronics ◽  
2021 ◽  
Vol 10 (2) ◽  
pp. 190
Author(s):  
Wu Ouyang ◽  
Zhigang Chen ◽  
Jia Wu ◽  
Genghua Yu ◽  
Heng Zhang
Keyword(s):  
Energy Consumption ◽  
Load Balancing ◽  
Markov Decision Process ◽  
Decision Process ◽  
Base Station ◽  
Edge Computing ◽  
Mobile Edge Computing ◽  
Task Migration ◽  
Migration Strategy ◽  
Markov Decision

As transportation becomes more convenient and efficient, users move faster and faster. When a user leaves the service range of the original edge server, the original edge server needs to migrate the tasks offloaded by the user to other edge servers. An effective task migration strategy needs to fully consider the location of users, the load status of edge servers, and energy consumption, which make designing an effective task migration strategy a challenge. In this paper, we innovatively proposed a mobile edge computing (MEC) system architecture consisting of multiple smart mobile devices (SMDs), multiple unmanned aerial vehicle (UAV), and a base station (BS). Moreover, we establish the model of the Markov decision process with unknown rewards (MDPUR) based on the traditional Markov decision process (MDP), which comprehensively considers the three aspects of the migration distance, the residual energy status of the UAVs, and the load status of the UAVs. Based on the MDPUR model, we propose a advantage-based value iteration (ABVI) algorithm to obtain the effective task migration strategy, which can help the UAV group to achieve load balancing and reduce the total energy consumption of the UAV group under the premise of ensuring user service quality. Finally, the results of simulation experiments show that the ABVI algorithm is effective. In particular, the ABVI algorithm has better performance than the traditional value iterative algorithm. And in a dynamic environment, the ABVI algorithm is also very robust.

Estimation of Path Travel Time Distributions in Stochastic Time-Varying Networks with Correlations

2021 ◽  
pp. 036119812110184
Author(s):  
Monika Filipovska ◽  
Hani S. Mahmassani ◽  
Archak Mittal
Keyword(s):  
Time Dependence ◽  
Travel Time ◽  
Correlation Structure ◽  
Time Varying ◽  
Travel Times ◽  
Trajectory Data ◽  
Time Varying Networks ◽  
Transportation Research ◽  
Search Approach ◽  
Stochastic Time

Transportation research has increasingly focused on the modeling of travel time uncertainty in transportation networks. From a user’s perspective, the performance of the network is experienced at the level of a path, and, as such, knowledge of variability of travel times along paths contemplated by the user is necessary. This paper focuses on developing approaches for the estimation of path travel time distributions in stochastic time-varying networks so as to capture generalized correlations between link travel times. Specifically, the goal is to develop methods to estimate path travel time distributions for any path in the networks by synthesizing available trajectory data from various portions of the path, and this paper addresses that problem in a two-fold manner. Firstly, a Monte Carlo simulation (MCS)-based approach is presented for the convolution of time-varying random variables with general correlation structures and distribution shapes. Secondly, a combinatorial data-mining approach is developed, which aims to utilize sparse trajectory data for the estimation of path travel time distributions by implicitly capturing the complex correlation structure in the network travel times. Numerical results indicate that the MCS approach allowing for time-dependence and a time-varying correlation structure outperforms other approaches, and that its performance is robust with respect to different path travel time distributions. Additionally, using the path segmentations from the segment search approach with a MCS approach with time-dependence also produces accurate and robust estimates of the path travel time distributions with the added benefit of shorter computation times.

Sign in / Sign up

Export Citation Format

Share Document