scholarly journals A Containerized Approach for Allocating Distributed Stream Queries to Fog Nodes

2021 ◽  
Author(s):  
Hamed Hasibi ◽  
Saeed Sedighian Kashi
Keyword(s):  
Fog Computing ◽  
Stream Processing ◽  
Stream Data ◽  
Process Data ◽  
Stream Query Processing ◽  
Tremendous Amount ◽  
Stream Processing Engines ◽  
Iot Devices ◽  
Distributed Stream Processing

Fog computing brings cloud capabilities closer to the Internet of Things (IoT) devices. IoT devices generate a tremendous amount of stream data towards the cloud via hierarchical fog nodes. To process data streams, many Stream Processing Engines (SPEs) have been developed. Without the fog layer, the stream query processing executes on the cloud, which forwards much traffic toward the cloud. When a hierarchical fog layer is available, a complex query can be divided into simple queries to run on fog nodes by using distributed stream processing. In this paper, we propose an approach to assign stream queries to fog nodes using container technology. We name this approach Stream Queries Placement in Fog (SQPF). Our goal is to minimize end-to-end delay to achieve a better quality of service. At first, in the emulation step, we make docker container instances from SPEs and evaluate their processing delay and throughput under different resource configurations and queries with varying input rates. Then in the placement step, we assign queries among fog nodes by using a genetic algorithm. The practical approach used in SQPF achieves a near-the-best assignment based on the lowest application deadline in real scenarios, and evaluation results are evidence of this goal.

scholarly journals A Containerized Approach for Allocating Distributed Stream Queries to Fog Nodes

2021 ◽  
Author(s):  
Hamed Hasibi ◽  
Saeed Sedighian Kashi
Keyword(s):  
Fog Computing ◽  
Stream Processing ◽  
Stream Data ◽  
Process Data ◽  
Stream Query Processing ◽  
Tremendous Amount ◽  
Stream Processing Engines ◽  
Iot Devices ◽  
Distributed Stream Processing

Fog computing brings cloud capabilities closer to the Internet of Things (IoT) devices. IoT devices generate a tremendous amount of stream data towards the cloud via hierarchical fog nodes. To process data streams, many Stream Processing Engines (SPEs) have been developed. Without the fog layer, the stream query processing executes on the cloud, which forwards much traffic toward the cloud. When a hierarchical fog layer is available, a complex query can be divided into simple queries to run on fog nodes by using distributed stream processing. In this paper, we propose an approach to assign stream queries to fog nodes using container technology. We name this approach Stream Queries Placement in Fog (SQPF). Our goal is to minimize end-to-end delay to achieve a better quality of service. At first, in the emulation step, we make docker container instances from SPEs and evaluate their processing delay and throughput under different resource configurations and queries with varying input rates. Then in the placement step, we assign queries among fog nodes by using a genetic algorithm. The practical approach used in SQPF achieves a near-the-best assignment based on the lowest application deadline in real scenarios, and evaluation results are evidence of this goal.

scholarly journals Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines

Electronics ◽  
2020 ◽  
Vol 9 (11) ◽  
pp. 1857
Author(s):  
Siwoon Son ◽  
Yang-Sae Moon
Keyword(s):  
Real World ◽  
Data Streams ◽  
Job Scheduling ◽  
Stream Processing ◽  
Process Data ◽  
Parallel Tasks ◽  
Stream Processing Engines ◽  
Job Scheduler ◽  
Distributed Stream Processing ◽  
Apache Storm

Distributed stream processing engines (DSPEs) deploy multiple tasks on distributed servers to process data streams in real time. Many DSPEs have provided locality-aware stream partitioning (LSP) methods to reduce network communication costs. However, an even job scheduler provided by DSPEs deploys tasks far away from each other on the distributed servers, which cannot use the LSP properly. In this paper, we propose a Locality/Fairness-aware job scheduler (L/F job scheduler) that considers locality together to solve problems of the even job scheduler that only considers fairness. First, the L/F job scheduler increases cohesion of contiguous tasks that require message transmissions for the locality. At the same time, it reduces coupling of parallel tasks that do not require message transmissions for the fairness. Next, we connect the contiguous tasks into a stream pipeline and evenly deploy stream pipelines to the distributed servers so that the L/F job scheduler achieves high cohesion and low coupling. Finally, we implement the proposed L/F job scheduler in Apache Storm, a representative DSPE, and evaluate it in both synthetic and real-world workloads. Experimental results show that the L/F job scheduler is similar in throughput compared to the even job scheduler, but latency is significantly improved by up to 139.2% for the LSP applications and by up to 140.7% even for the non-LSP applications. The L/F job scheduler also improves latency by 19.58% and 12.13%, respectively, in two real-world workloads. These results indicate that our L/F job scheduler provides superior processing performance for the DSPE applications.

Survey on DDoS Attacks and Defense Mechanisms in Cloud and Fog Computing

2018 ◽  
Vol 10 (3) ◽  
pp. 61-83 ◽  
Author(s):  
Deepali Chaudhary ◽  
Kriti Bhushan ◽  
B.B. Gupta
Keyword(s):  
Cloud Computing ◽  
Defense Mechanisms ◽  
Low Cost ◽  
Fog Computing ◽  
Smart Devices ◽  
Cloud Environment ◽  
Security Issue ◽  
Process Data ◽  
Ddos Attack ◽  
Iot Devices

This article describes how cloud computing has emerged as a strong competitor against traditional IT platforms by offering low-cost and “pay-as-you-go” computing potential and on-demand provisioning of services. Governments, as well as organizations, have migrated their entire or most of the IT infrastructure to the cloud. With the emergence of IoT devices and big data, the amount of data forwarded to the cloud has increased to a huge extent. Therefore, the paradigm of cloud computing is no longer sufficient. Furthermore, with the growth of demand for IoT solutions in organizations, it has become essential to process data quickly, substantially and on-site. Hence, Fog computing is introduced to overcome these drawbacks of cloud computing by bringing intelligence to the edge of the network using smart devices. One major security issue related to the cloud is the DDoS attack. This article discusses in detail about the DDoS attack, cloud computing, fog computing, how DDoS affect cloud environment and how fog computing can be used in a cloud environment to solve a variety of problems.

Reliable stream data processing for elastic distributed stream processing systems

2019 ◽  
Vol 23 (2) ◽  
pp. 555-574 ◽  
Author(s):  
Xiaohui Wei ◽  
Yuan Zhuang ◽  
Hongliang Li ◽  
Zhiliang Liu
Keyword(s):  
Data Processing ◽  
Stream Processing ◽  
Stream Data ◽  
Stream Data Processing ◽  
Distributed Stream Processing

scholarly journals Construction Scheme of a Scalable Distributed Stream Processing Infrastructure Using Ray and Apache Kafka

10.29007/8lbk ◽  
2019 ◽  
Author(s):  
Kasumi Kato ◽  
Atsuko Takefusa ◽  
Hidemoto Nakada ◽  
Masato Oguchi
Keyword(s):  
Stream Processing ◽  
Sensor Data ◽  
Prototype System ◽  
Stream Data ◽  
Computing Power ◽  
Moving Image ◽  
Construction Scheme ◽  
Large Numbers ◽  
Distributed Stream Processing ◽  
And Storage

The spread of various sensors and the development of cloud computing technologies en- able the accumulation and use of large numbers of live logs in ordinary homes. To operate a service that utilizes sensor data, it is difficult to install servers and storage in ordinary homes and to analyze the collected data from sensors. Those data are typically transmitted from sensors to a cloud and analyzed in the cloud. However, services that involve moving image analysis must transfer large amounts of data continuously and require high computing power for analysis. Hence, it is highly difficult to process them in real time in the cloud using a conventional stream data processing framework. In this research, we propose a construction scheme for a highly efficient distributed stream processing infrastructure that enables scalable processing of moving image recognition tasks according to the amount of data that are transmitted from sensors. We implement a prototype system of the proposed distributed stream processing infrastructure using Ray and Apache Kafka, which is a distributed messaging system, and we evaluate its performance. The experimental results demonstrate that the proposed distributed stream processing infrastructure is highly scalable.

Survey on DDoS Attacks and Defense Mechanisms in Cloud and Fog Computing

2019 ◽  
pp. 1927-1951
Author(s):  
Deepali Chaudhary ◽  
Kriti Bhushan ◽  
B.B. Gupta
Keyword(s):  
Cloud Computing ◽  
Defense Mechanisms ◽  
Low Cost ◽  
Fog Computing ◽  
Smart Devices ◽  
Cloud Environment ◽  
Security Issue ◽  
Process Data ◽  
Ddos Attack ◽  
Iot Devices

This article describes how cloud computing has emerged as a strong competitor against traditional IT platforms by offering low-cost and “pay-as-you-go” computing potential and on-demand provisioning of services. Governments, as well as organizations, have migrated their entire or most of the IT infrastructure to the cloud. With the emergence of IoT devices and big data, the amount of data forwarded to the cloud has increased to a huge extent. Therefore, the paradigm of cloud computing is no longer sufficient. Furthermore, with the growth of demand for IoT solutions in organizations, it has become essential to process data quickly, substantially and on-site. Hence, Fog computing is introduced to overcome these drawbacks of cloud computing by bringing intelligence to the edge of the network using smart devices. One major security issue related to the cloud is the DDoS attack. This article discusses in detail about the DDoS attack, cloud computing, fog computing, how DDoS affect cloud environment and how fog computing can be used in a cloud environment to solve a variety of problems.

Fog Computing Qos Review and Open Challenges

2021 ◽  
pp. 1147-1157
Author(s):  
R. Babu ◽  
K. Jayashree ◽  
R. Abirami
Keyword(s):  
Programming Model ◽  
Fog Computing ◽  
Big Data Analytics ◽  
Security And Privacy ◽  
Computation Offloading ◽  
Future Research ◽  
Future Research Directions ◽  
Iot Devices ◽  
Research Challenge

Internet of Things (IoT) enables inters connectivity among devices and platforms. IoT devices such as sensors, or embedded systems offer computational, storage, and networking resources and the existence of these resources permits to move the execution of IoT applications to the edge of the network and it is known as fog computing. It is able to handle billions of Internet-connected devices and is well situated for real-time big data analytics and provides advantages in advertising and personal computing. The main issues in fog computing includes fog networking, QoS, interfacing and programming model, computation offloading, accounting, billing and monitoring, provisioning and resource management, security and privacy. A particular research challenge is the Quality of Service metric for fog services. Thus, this paper gives a survey of cloud computing, discusses the QoS metrics, and the future research directions in fog computing.

Sign in / Sign up

Export Citation Format

Share Document