Resource aware scheduler for distributed stream processing in cloud native environments

2021 ◽  
Author(s):  
Madushi Sarathchandra ◽  
Chulani Karandana ◽  
Winma Heenatigala ◽  
Miyuru Dayarathna ◽  
Sanath Jayasena
Keyword(s):  
Stream Processing ◽  
Resource Aware ◽  
Distributed Stream Processing

Predictable remote invocations for distributed stream processing

2020 ◽  
Vol 107 ◽  
pp. 716-729 ◽  
Author(s):  
P. Basanta-Val ◽  
N. Fernández-García ◽  
L. Sánchez-Fernández
Keyword(s):  
Stream Processing ◽  
Distributed Stream Processing

Evaluating Fault Tolerance of Distributed Stream Processing Systems

2020 ◽  
pp. 101-116
Author(s):  
Xiaotong Wang ◽  
Cheng Jiang ◽  
Junhua Fang ◽  
Ke Shu ◽  
Rong Zhang ◽  
...  
Keyword(s):  
Fault Tolerance ◽  
Stream Processing ◽  
Distributed Stream Processing

scholarly journals A Containerized Approach for Allocating Distributed Stream Queries to Fog Nodes

2021 ◽  
Author(s):  
Hamed Hasibi ◽  
Saeed Sedighian Kashi
Keyword(s):  
Fog Computing ◽  
Stream Processing ◽  
Stream Data ◽  
Process Data ◽  
Stream Query Processing ◽  
Tremendous Amount ◽  
Stream Processing Engines ◽  
Iot Devices ◽  
Distributed Stream Processing

Fog computing brings cloud capabilities closer to the Internet of Things (IoT) devices. IoT devices generate a tremendous amount of stream data towards the cloud via hierarchical fog nodes. To process data streams, many Stream Processing Engines (SPEs) have been developed. Without the fog layer, the stream query processing executes on the cloud, which forwards much traffic toward the cloud. When a hierarchical fog layer is available, a complex query can be divided into simple queries to run on fog nodes by using distributed stream processing. In this paper, we propose an approach to assign stream queries to fog nodes using container technology. We name this approach Stream Queries Placement in Fog (SQPF). Our goal is to minimize end-to-end delay to achieve a better quality of service. At first, in the emulation step, we make docker container instances from SPEs and evaluate their processing delay and throughput under different resource configurations and queries with varying input rates. Then in the placement step, we assign queries among fog nodes by using a genetic algorithm. The practical approach used in SQPF achieves a near-the-best assignment based on the lowest application deadline in real scenarios, and evaluation results are evidence of this goal.

Sign in / Sign up

Export Citation Format

Share Document