Big Data solutions - data ingestion and stream processing for demand response management

2019 ◽  
Author(s):  
Simona-Vasilica Oprea ◽  
Adela Bara ◽  
Vlad Diaconita ◽  
Dan Preotescu ◽  
Osman Bulent Tor
Keyword(s):  
Big Data ◽  
Demand Response ◽  
Stream Processing ◽  
Data Ingestion

scholarly journals Stream Processing Languages in the Big Data Era

2018 ◽  
Vol 47 (2) ◽  
pp. 29-40 ◽  
Author(s):  
Martin Hirzel ◽  
Guillaume Baudart ◽  
Angela Bonifati ◽  
Emanuele Della Valle ◽  
Sherif Sakr ◽  
...  
Keyword(s):  
Big Data ◽  
Stream Processing

scholarly journals A Systematic Study for Big Data Stream Processing Frameworks

2016 ◽  
Vol 2 (2) ◽  
pp. 4
Author(s):  
Ali Yazici ◽  
Ziya Karakaya ◽  
Mohammed Alayyoub
Keyword(s):  
Big Data ◽  
Systematic Study ◽  
Data Stream ◽  
Stream Processing ◽  
Systematic Mapping ◽  
Related Research ◽  
Research Outcomes ◽  
Research Questions ◽  
State Of Art ◽  
Processing Framework

The choice of the most effective stream processing framework (SPF) for Big Data has been an important issue among the researchers and practioners. Each of the SPFs has different cutting edge technologies in their steps of processing the data in motion that gives them a better advantage over the others. Even though, these technologies used in each SPF might better them, it is still difficult to conclude which framework berforms better under different scenarios and conditions. In this paper, we aim to show trends and differences about several SPFs for Big Data by using the so called Systematic Mapping (SM) approach using the related research outcomes. To achieve this objective, nine research questions (RQs) were raised, in which 91 studies that were conducted between 2010 and 2015 were evaluated. We present the trends by classifying the research on SPFs with respect to the proposed RQs which can direct researchers in getting an state-of-art overview of the field.

scholarly journals SIDELOADING – INGESTION OF LARGE POINT CLOUDS INTO THE APACHE SPARK BIG DATA ENGINE

2016 ◽  
Vol XLI-B2 ◽  
pp. 343-348
Author(s):  
J. Boehm ◽  
K. Liu ◽  
C. Alis
Keyword(s):  
Big Data ◽  
Point Cloud ◽  
Point Clouds ◽  
Geospatial Data ◽  
Apache Spark ◽  
Cloud Data ◽  
Binary File ◽  
Data Framework ◽  
File Formats ◽  
Data Ingestion

In the geospatial domain we have now reached the point where data volumes we handle have clearly grown beyond the capacity of most desktop computers. This is particularly true in the area of point cloud processing. It is therefore naturally lucrative to explore established big data frameworks for big geospatial data. The very first hurdle is the import of geospatial data into big data frameworks, commonly referred to as data ingestion. Geospatial data is typically encoded in specialised binary file formats, which are not naturally supported by the existing big data frameworks. Instead such file formats are supported by software libraries that are restricted to single CPU execution. We present an approach that allows the use of existing point cloud file format libraries on the Apache Spark big data framework. We demonstrate the ingestion of large volumes of point cloud data into a compute cluster. The approach uses a map function to distribute the data ingestion across the nodes of a cluster. We test the capabilities of the proposed method to load billions of points into a commodity hardware compute cluster and we discuss the implications on scalability and performance. The performance is benchmarked against an existing native Apache Spark data import implementation.

Sign in / Sign up

Export Citation Format

Share Document