scholarly journals A Time-efficient Multi-Protocol Probe Scheme for Fine-grain IoT Device Identification

Sensors ◽  
2020 ◽  
Vol 20 (7) ◽  
pp. 1863
Author(s):  
Dan Yu ◽  
Peiyang Li ◽  
Yongle Chen ◽  
Yao Ma ◽  
Junjie Chen
Keyword(s):  
Identification Accuracy ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Device Identification ◽  
Sequence Segment ◽  
Fine Grain ◽  
Sample Data ◽  
Different Types ◽  
Markov Decision ◽  
Iot Devices

Internet of Things (IoT) devices connected to the Internet are exploding, which poses a significant threat for their management and security protection. IoT device identification is a prerequisite for discovering, monitoring, and protecting these devices. Although we can identify the device type easily through grabbing protocol banner information, both brand and model of different types of device are various and diverse. We should therefore utilize multi-protocol probes to improve the fineness of device identification and obtain the corresponding brand and model. However, it is still a challenge to balance between the multi-protocol probe overhead and the identification fineness. To solve this problem, we proposed a time-efficient multi-protocol probe scheme for fine-grain devices identification. We first adopted the concept of reinforcement learning to model the banner-based device identification process into a Markov decision process (MDP). Through the value iteration algorithm, an optimal multi-protocol probe sequence is generated for a type-known IoT device, and then the optimal multi-protocol probes sequence segment is extracted based on the gain threshold of identification accuracy. We took 132,835 webcams as the sample data to experiment. The experimental results showed that our optimal multi-protocol probes sequence segment could reduce the identification time of webcams’ brand and model by 50.76% and achieve the identification accuracy of 90.5% and 92.3% respectively. In addition, we demonstrated that our time-efficient optimal multi-protocol probe scheme could also significantly improve the identification efficiency of other IoT devices, such as routers and printers.

Adiabatic Markov Decision Process: Convergence of Value Iteration Algorithm

2016 ◽  
Vol 138 (6) ◽  
Author(s):  
Thai Duong ◽  
Duong Nguyen-Huu ◽  
Thinh Nguyen
Keyword(s):  
Markov Decision Process ◽  
Decision Process ◽  
Transition Probability ◽  
Transition Probability Matrix ◽  
Rate Of Change ◽  
Optimal Decision ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Markov Decision ◽  
Value Iteration Algorithm

Markov decision process (MDP) is a well-known framework for devising the optimal decision-making strategies under uncertainty. Typically, the decision maker assumes a stationary environment which is characterized by a time-invariant transition probability matrix. However, in many real-world scenarios, this assumption is not justified, thus the optimal strategy might not provide the expected performance. In this paper, we study the performance of the classic value iteration algorithm for solving an MDP problem under nonstationary environments. Specifically, the nonstationary environment is modeled as a sequence of time-variant transition probability matrices governed by an adiabatic evolution inspired from quantum mechanics. We characterize the performance of the value iteration algorithm subject to the rate of change of the underlying environment. The performance is measured in terms of the convergence rate to the optimal average reward. We show two examples of queuing systems that make use of our analysis framework.

A Modified Value Iteration Algorithm for Discounted Markov Decision Processes

2015 ◽  
Vol 13 (3) ◽  
pp. 47-57 ◽  
Author(s):  
Sanaa Chafik ◽  
Cherki Daoui
Keyword(s):  
Markov Decision Processes ◽  
Decision Processes ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Decomposition Technique ◽  
Artificial Data ◽  
Markov Decision ◽  
Speed Up ◽  
Value Iteration Algorithm

As many real applications need a large amount of states, the classical methods are intractable for solving large Markov Decision Processes. The decomposition technique basing on the topology of each state in the associated graph and the parallelization technique are very useful methods to cope with this problem. In this paper, the authors propose a Modified Value Iteration algorithm, adding the parallelism technique. They test their implementation on artificial data using an Open MP that offers a significant speed-up.

scholarly journals Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

2005 ◽  
Vol 42 (4) ◽  
pp. 905-918 ◽  
Author(s):  
Rolando Cavazos-Cadena ◽  
Raúl Montes-De-Oca
Keyword(s):  
Iteration Algorithm ◽  
Value Iteration ◽  
Stationary Policy ◽  
Long Run ◽  
Risk Sensitive ◽  
Finite State ◽  
Markov Decision ◽  
Optimal Stationary Policy ◽  
Compact Action Sets ◽  
Nonstationary Value Iteration

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

scholarly journals An Optimal Initial Radio Access Technology Selection Method for Heterogeneous Wireless Networks

2021 ◽  
Author(s):  
Mizanur Rahman
Keyword(s):  
Wireless Networks ◽  
Heterogeneous Wireless Networks ◽  
Geographical Area ◽  
Technology Selection ◽  
System Capacity ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Radio Access Technology ◽  
Radio Access ◽  
Markov Decision

In Heterogeneous Wireless Networks, different overlapped Radio Access Technologies (RATs) can coexist with each other in the same geographical area. In such environment, a challenge is to select in which available RATs a user can be connected upon making an incoming service request. In this thesis, this challenge is investigated by proposing a Joint Call Admission Control (JCAC) -based approach that uses the framework of Semi-Markov Decision Process for initial RAT selection in two co-located wireless networks supporting two different service classes. The optimization problem involves the design of a cost function that weights the blocking cost and the energy consumption cost. The JCAC optimal policy is derived using the Value Iteration Algorithm. Simulations results show that the system capacity is maximized while selecting the less energy consuming RAT.

scholarly journals Perception-Aware Point-Based Value Iteration for Partially Observable Markov Decision Processes

2019 ◽  
Author(s):  
Mahsa Ghasemi ◽  
Ufuk Topcu
Keyword(s):  
Markov Decision Processes ◽  
Active Role ◽  
Decision Processes ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Greedy Strategy ◽  
Markov Decision ◽  
Partially Observable Markov ◽  
Observation Selection ◽  
Partially Observable

In conventional partially observable Markov decision processes, the observations that the agent receives originate from fixed known distributions. However, in a variety of real-world scenarios, the agent has an active role in its perception by selecting which observations to receive. We avoid combinatorial expansion of the action space from integration of planning and perception decisions, through a greedy strategy for observation selection that minimizes an information-theoretic measure of the state uncertainty. We develop a novel point-based value iteration algorithm that incorporates this greedy strategy to pick perception actions for each sampled belief point in each iteration. As a result, not only the solver requires less belief points to approximate the reachable subspace of the belief simplex, but it also requires less computation per iteration. Further, we prove that the proposed algorithm achieves a near-optimal guarantee on value function with respect to an optimal perception strategy, and demonstrate its performance empirically.

scholarly journals Perseus: Randomized Point-based Value Iteration for POMDPs

2005 ◽  
Vol 24 ◽  
pp. 195-220 ◽  
Author(s):  
M. T.J. Spaan ◽  
N. Vlassis
Keyword(s):  
Large Scale ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Planning Under Uncertainty ◽  
Markov Decision ◽  
Finite Set ◽  
Partially Observable ◽  
Set Of Points ◽  
Action Spaces ◽  
Belief Set

Partially observable Markov decision processes (POMDPs) form an attractive and principled framework for agent planning under uncertainty. Point-based approximate techniques for POMDPs compute a policy based on a finite set of points collected in advance from the agent's belief space. We present a randomized point-based value iteration algorithm called Perseus. The algorithm performs approximate value backup stages, ensuring that in each backup stage the value of each point in the belief set is improved; the key observation is that a single backup may improve the value of many belief points. Contrary to other point-based methods, Perseus backs up only a (randomly selected) subset of points in the belief set, sufficient for improving the value of each belief point in the set. We show how the same idea can be extended to dealing with continuous action spaces. Experimental results show the potential of Perseus in large scale POMDP problems.

scholarly journals A Semi-Markov decision model-based brokering mechanism for mobile cloud market

2021 ◽  
Author(s):  
Elena Degtiareve
Keyword(s):  
Service Providers ◽  
Cloud Service ◽  
Cloud Services ◽  
Iteration Algorithm ◽  
Value Iteration ◽  
Security Risk ◽  
System Cost ◽  
Cloud Broker ◽  
Markov Decision ◽  
Markov Decision Model

As the multitude and complexity of cloud market increases the evaluation and selection of cloud services becomes a burdensome task for the users. With the increased rise of available services from various Cloud Service Providers (CSP), the role of cloud brokers becomes more and more important. In this thesis, the challenge of optimally allocating multiple cloud system resources to multiple mobile user’s requests with different requirements is investigated and an optimal Cloud Broker model is proposed. The cloud brokering mechanism is formulated as a Semi-Markov Decision Process (SMDP) model under the average system cost criteria, taking into consideration the cost of the occupying computing resources, the communication costs, the request traffic, and some security risk degrees and resource requirements from the multiple mobile users. Through minimizing the overall system cost, the optimal resource allocation policy is derived by using the Value Iteration Algorithm. Simulation results are provided, demonstrating the efficiency of the proposed Cloud Broker design.

Sign in / Sign up

Export Citation Format

Share Document