Comparative Analysis of Apache Spark and Hadoop MapReduce Using Various Parameters and Execution Time

2020 ◽  
pp. 719-725
Author(s):  
Bhagavathula Meena ◽  
I. S. L. Sarwani ◽  
M. Archana ◽  
P. Supriya
Keyword(s):  
Comparative Analysis ◽  
Execution Time ◽  
Apache Spark ◽  
Hadoop Mapreduce

DISTRIBUTED PROCESSING OF LARGE VOLUMES OF TRANSACTIONAL DATA

2020 ◽  
pp. 27-36
Author(s):  
O. Dmytriieva ◽  
◽  
D. Nikulin
Keyword(s):  
Distributed Processing ◽  
Apache Spark ◽  
Hadoop Mapreduce ◽  
Transactional Data

Роботу присвячено питанням розподіленої обробки транзакцій при проведенні аналізу великих обсягів даних з метою пошуку асоціативних правил. На основі відомих алгоритмів глибинного аналізу даних для пошуку частих предметних наборів AIS та Apriori було визначено можливі варіанти паралелізації, які позбавлені необхідності ітераційного сканування бази даних та великого споживання пам'яті. Досліджено можливість перенесення обчислень на різні платформи, які підтримують паралельну обробку даних. В якості обчислювальних платформ було обрано MapReduce – потужну базу для обробки великих, розподілених наборів даних на кластері Hadoop, а також програмний інструмент для обробки надзвичайно великої кількості даних Apache Spark. Проведено порівняльний аналіз швидкодії розглянутих методів, отримано рекомендації щодо ефективного використання паралельних обчислювальних платформ, запропоновано модифікації алгоритмів пошуку асоціативних правил. В якості основних завдань, реалізованих в роботі, слід визначити дослідження сучасних засобів розподіленої обробки структурованих і не структурованих даних, розгортання тестового кластера в хмарному сервісі, розробку скриптів для автоматизації розгортання кластера, проведення модифікацій розподілених алгоритмів з метою адаптації під необхідні фреймворки розподілених обчислень, отримання показників швидкодії обробки даних в послідовному і розподіленому режимах з застосуванням Hadoop MapReduce. та Apache Spark, проведення порівняльного аналізу результатів тестових вимірів швидкодії, отримання та обґрунтування залежності між кількістю оброблюваних даних, і часом, витраченим на обробку, оптимізацію розподілених алгоритмів пошуку асоціативних правил при обробці великих обсягів транзакційних даних, отримання показників швидкодії розподіленої обробки існуючими програмними засобами. Ключові слова: розподілена обробка, транзакційні дані, асоціативні правила, обчислюваний кластер, Hadoop, MapReduce, Apache Spark

scholarly journals Multi-Aspect Incremental Tensor Decomposition Based on Distributed In-Memory Big Data Systems

2020 ◽  
Vol 5 (2) ◽  
pp. 13-32
Author(s):  
Hye-Kyung Yang ◽  
Hwan-Seung Yong
Keyword(s):  
Execution Time ◽  
Three Dimensional ◽  
Large Data ◽  
Tensor Decomposition ◽  
Decomposition Methods ◽  
Apache Spark ◽  
Decomposition Algorithms ◽  
Data Systems ◽  
Big Data Systems ◽  
Spark Framework

AbstractPurposeWe propose InParTen2, a multi-aspect parallel factor analysis three-dimensional tensor decomposition algorithm based on the Apache Spark framework. The proposed method reduces re-decomposition cost and can handle large tensors.Design/methodology/approachConsidering that tensor addition increases the size of a given tensor along all axes, the proposed method decomposes incoming tensors using existing decomposition results without generating sub-tensors. Additionally, InParTen2 avoids the calculation of Khari–Rao products and minimizes shuffling by using the Apache Spark platform.FindingsThe performance of InParTen2 is evaluated by comparing its execution time and accuracy with those of existing distributed tensor decomposition methods on various datasets. The results confirm that InParTen2 can process large tensors and reduce the re-calculation cost of tensor decomposition. Consequently, the proposed method is faster than existing tensor decomposition algorithms and can significantly reduce re-decomposition cost.Research limitationsThere are several Hadoop-based distributed tensor decomposition algorithms as well as MATLAB-based decomposition methods. However, the former require longer iteration time, and therefore their execution time cannot be compared with that of Spark-based algorithms, whereas the latter run on a single machine, thus limiting their ability to handle large data.Practical implicationsThe proposed algorithm can reduce re-decomposition cost when tensors are added to a given tensor by decomposing them based on existing decomposition results without re-decomposing the entire tensor.Originality/valueThe proposed method can handle large tensors and is fast within the limited-memory framework of Apache Spark. Moreover, InParTen2 can handle static as well as incremental tensor decomposition.

scholarly journals The comparative analysis of modern ETL tools

2021 ◽  
Vol 19 ◽  
pp. 126-131
Author(s):  
Vitalii Mayuk ◽  
Ivan Falchuk ◽  
Piotr Muryjas
Keyword(s):  
Comparative Analysis ◽  
Data Warehouse ◽  
Execution Time ◽  
Analytical Data ◽  
Time Efficiency ◽  
Transactional Data ◽  
Research Efficiency

Each data warehouse requires loading properly processed transactional data. The process that performs this task is known as extract-transform-load (ETL). The efficiency of its implementation affects how quickly the user will have the access to the current analytical data. The paper presents the results of research efficiency of ETL performance of its stage with the use of Azure Synapse (AS) and Azure Data Factory (ADF). The research included selection, sorting and aggregating data, joining tables, and loading data into target tables. To evaluate the efficiency of these operations, the criterion of their execution time has been used. The obtained results indicate that the ADF tool provides a much higher time efficiency of loading transactional data into the data warehouse comparing to AS.

A Study on the Performance and Scalability of Apache Flink Over Hadoop MapReduce

2019 ◽  
Vol 2 (1) ◽  
pp. 61-73
Author(s):  
Pankaj Lathar ◽  
K. G. Srinivasa
Keyword(s):  
Comparative Analysis ◽  
Parallel Processing ◽  
Data Processing ◽  
Real World ◽  
Memory Management ◽  
Programming Model ◽  
Raw Data ◽  
Hadoop Mapreduce ◽  
Active Memory ◽  
Real World Problems

With the advancements in science and technology, data is being generated at a staggering rate. The raw data generated is generally of high value and may conceal important information with the potential to solve several real-world problems. In order to extract this information, the raw data available must be processed and analysed efficiently. It has however been observed, that such raw data is generated at a rate faster than it can be processed by traditional methods. This has led to the emergence of the popular parallel processing programming model – MapReduce. In this study, the authors perform a comparative analysis of two popular data processing engines – Apache Flink and Hadoop MapReduce. The analysis is based on the parameters of scalability, reliability and efficiency. The results reveal that Flink unambiguously outperformance Hadoop's MapReduce. Flink's edge over MapReduce can be attributed to following features – Active Memory Management, Dataflow Pipelining and an Inline Optimizer. It can be concluded that as the complexity and magnitude of real time raw data is continuously increasing, it is essential to explore newer platforms that are adequately and efficiently capable of processing such data.

Applying compression algorithms on hadoop cluster implementing through apache tez and hadoop mapreduce

2018 ◽  
Vol 7 (2.26) ◽  
pp. 80
Author(s):  
Dr E. Laxmi Lydia ◽  
M Srinivasa Rao
Keyword(s):  
Big Data ◽  
Execution Time ◽  
Big Data Analytics ◽  
Research Area ◽  
Word Count ◽  
Hadoop Mapreduce ◽  
Interactive Query ◽  
Hadoop Distributed File System ◽  
Hadoop Cluster ◽  
Compressed Data

The latest and famous subject all over the cloud research area is Big Data; its main appearances are volume, velocity and variety. The characteristics are difficult to manage through traditional software and their various available methodologies. To manage the data which is occurring from various domains of big data are handled through Hadoop, which is open framework software which is mainly developed to provide solutions. Handling of big data analytics is done through Hadoop Map Reduce framework and it is the key engine of hadoop cluster and it is extensively used in these days. It uses batch processing system.Apache developed an engine named "Tez", which supports interactive query system and it won't writes any temporary data into the Hadoop Distributed File System(HDFS).The paper mainly focuses on performance juxtaposition of MapReduce and TeZ, performance of these two engines are examined through the compression of input files and map output files. To compare two engines we used Bzip compression algorithm for the input files and snappy for the map out files. Word Count and Terasort gauge are used on our experiments. For the Word Count gauge, the results shown that Tez engine has better execution time than Hadoop MapReduce engine for the both compressed and non-compressed data. It has reduced the execution time nearly 39% comparing to the execution time of the Hadoop MapReduce engine. Correspondingly for the terasort gauge, the Tez engine has higher execution time than Hadoop MapReduce engine.  

scholarly journals Performance Evaluation of a Big Data Application on Apache Spark

2021 ◽  
Author(s):  
Jeanne Alcantara
Keyword(s):  
Big Data ◽  
Performance Evaluation ◽  
Execution Time ◽  
Apache Spark ◽  
Massive Data ◽  
Application Performance ◽  
Data Application ◽  
Size Number ◽  
Big Data Application ◽  
The Impact

Apache Spark enables a big data application—one that takes massive data as input and may produce massive data along its execution—to run in parallel on multiple nodes. Hence, for a big data application, performance is a vital issue. This project analyzes a WordCount application using Apache Spark, where the impact on the execution time and average utilization is assessed. To facilitate this assessment, the number of executor cores and the size of executor memory are varied across different sizes of data that the application has to process, and the different number of nodes in the cluster that the application runs on. It is concluded that different pairs (data size, number of nodes in the cluster) require different number of executor cores and different size of executor memory to obtain optimum results for execution time and average node utilization.

scholarly journals Performance Evaluation of a Big Data Application on Apache Spark

2021 ◽  
Author(s):  
Jeanne Alcantara
Keyword(s):  
Big Data ◽  
Performance Evaluation ◽  
Execution Time ◽  
Apache Spark ◽  
Massive Data ◽  
Application Performance ◽  
Data Application ◽  
Size Number ◽  
Big Data Application ◽  
The Impact

Apache Spark enables a big data application—one that takes massive data as input and may produce massive data along its execution—to run in parallel on multiple nodes. Hence, for a big data application, performance is a vital issue. This project analyzes a WordCount application using Apache Spark, where the impact on the execution time and average utilization is assessed. To facilitate this assessment, the number of executor cores and the size of executor memory are varied across different sizes of data that the application has to process, and the different number of nodes in the cluster that the application runs on. It is concluded that different pairs (data size, number of nodes in the cluster) require different number of executor cores and different size of executor memory to obtain optimum results for execution time and average node utilization.

Big Data Processing on Cloud Computing Using Hadoop Mapreduce and Apache Spark

2018 ◽  
pp. 224-250
Author(s):  
Yassir Samadi ◽  
Mostapha Zbakh ◽  
Amine Haouari
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Data Processing ◽  
Apache Spark ◽  
Big Data Processing ◽  
Data Intensive ◽  
Hadoop Mapreduce ◽  
Huge Data ◽  
Increasing Demand ◽  
Exponential Rates

Size of the data used by enterprises has been growing at exponential rates since last few years; handling such huge data from various sources is a challenge for Businesses. In addition, Big Data becomes one of the major areas of research for Cloud Service providers due to a large amount of data produced every day, and the inefficiency of traditional algorithms and technologies to handle these large amounts of data. In order to resolve the aforementioned problems and to meet the increasing demand for high-speed and data-intensive computing, several solutions have been developed by researches and developers. Among these solutions, there are Cloud Computing tools such as Hadoop MapReduce and Apache Spark, which work on the principles of parallel computing. This chapter focuses on how big data processing challenges can be handled by using Cloud Computing frameworks and the importance of using Cloud Computing by businesses

scholarly journals Multicriteria assessment of selected types of ceilings

2018 ◽  
Vol 219 ◽  
pp. 04011
Author(s):  
Julia Nowak ◽  
Edyta Plebankiewicz
Keyword(s):  
Comparative Analysis ◽  
Thermal Insulation ◽  
Execution Time ◽  
Lower Surface ◽  
Sound Insulation ◽  
Multicriteria Assessment ◽  
Comparison Criteria

The paper compares three systems of the rib-and-slab ceiling: Vector, Rector and Teriva. The following comparison criteria were selected: the tare weight of ceiling beams, execution time, slab faulting, sound insulation, thermal insulation and aesthetics of the lower surface. The results of the multicriteria comparative analysis clearly indicated the advantage of the Vector ceiling, assuming the same weights for all criteria taken into account.

Sign in / Sign up

Export Citation Format

Share Document