Treatment and Research of Massive Data Mining Based on Cloud Computing

2013 ◽  
Vol 765-767 ◽  
pp. 941-944
Author(s):  
Peng Wang ◽  
Jia Nan Wang ◽  
Ji Ci Ba ◽  
Yu Tan
Keyword(s):  
Data Mining ◽  
Cloud Computing ◽  
Data Processing ◽  
Programming Model ◽  
Massive Data ◽  
Distributed Data ◽  
Distributed Data Processing ◽  
Hadoop Platform ◽  
Core Framework

This paper introduces SPRINT algorithm optimized in the Hadoop core framework. Combing the data mining process, we will study the cloud computing in the MapReduce programming model, then improve and optimize the SPRINT algorithm in conjunction with the mode, transplant the optimized algorithm to Hadoop platform for distributed data processing.

scholarly journals Apache Nemo: A Framework for Optimizing Distributed Data Processing

2020 ◽  
Vol 38 (3-4) ◽  
pp. 1-31
Author(s):  
Won Wook Song ◽  
Youngseok Yang ◽  
Jeongyoon Eo ◽  
Jangho Seo ◽  
Joo Yeon Kim ◽  
...  
Keyword(s):  
Data Processing ◽  
High Performance ◽  
Programming Model ◽  
Compiler Optimization ◽  
Ease Of Use ◽  
Distributed Data ◽  
Performance Improvements ◽  
Distributed Data Processing ◽  
Fine Control ◽  
High Level

Optimizing scheduling and communication of distributed data processing for resource and data characteristics is crucial for achieving high performance. Existing approaches to such optimizations largely fall into two categories. First, distributed runtimes provide low-level policy interfaces to apply the optimizations, but do not ensure the maintenance of correct application semantics and thus often require significant effort to use. Second, policy interfaces that extend a high-level application programming model ensure correctness, but do not provide sufficient fine control. We describe Apache Nemo, an optimization framework for distributed dataflow processing that provides fine control for high performance and also ensures correctness for ease of use. We combine several techniques to achieve this, including an intermediate representation of dataflow, compiler optimization passes, and runtime extensions. Our evaluation results show that Nemo enables composable and reusable optimizations that bring performance improvements on par with existing specialized runtimes tailored for a specific deployment scenario. Apache Nemo is open-sourced at https://nemo.apache.org as an Apache incubator project.

scholarly journals Uma análise do consumo de energia de ambientes de processamento de dados massivos em nuvem

2018 ◽  
Author(s):  
Nestor D. O. Volpini ◽  
Vinicius S. Conceição ◽  
Raphael L. Pontes ◽  
Dorgival Guedes
Keyword(s):  
Data Mining ◽  
Cloud Computing ◽  
Big Data ◽  
Data Processing ◽  
Performance Metrics ◽  
Massive Data ◽  
Distributed Environments ◽  
Big Data Applications ◽  
Massive Data Processing ◽  
Data Mining Application

Massive data processing (big-data) related fields and cloud computing have been growing conjointly. Thus, data processing is among the largest resource consumers in datacenters, consuming around 2% of global energy. Comprehension of how elements such as virtualized environments and applications' parallelization degree affect such consumption is therefore an urgent need. This article relies on a monitoring solution that provides performance metrics, data mining application logs, and data produced in distributed environments to assess how power consumption of virtualized big-data applications varies on allocated resources.

scholarly journals DolphinNext: a distributed data processing platform for high throughput genomics

BMC Genomics ◽  
2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Onur Yukselen ◽  
Osman Turkyilmaz ◽  
Ahmet Rasit Ozturk ◽  
Manuel Garber ◽  
Alper Kucukural
Keyword(s):  
Data Processing ◽  
High Throughput ◽  
Distributed Data ◽  
Distributed Data Processing ◽  
Processing Platform

Was ist Computerleistung am Arbeitsplatz?

1979 ◽  
Vol 21 (2) ◽  
Author(s):  
L. J. Heinrich
Keyword(s):  
Data Processing ◽  
Distributed Data ◽  
Distributed Data Processing

Der Beitrag erläutert das subjektive Verständnis des Begriffes ,,Computerleistung am Arbeitsplatz" als Schlagwort für eine progressive Gestaltungsphilosophie computergestützter Informationssysteme. Sie impliziert sowohl die Anwendung moderner Hard- und Softwaretechnologien, wie sie für die 80er Jahre bestimmend sein werden, als auch die in den Vordergrund rückende Berücksichtigung der sowohl von der Arbeitsaufgabe bestimmten als auch der subjektiven Benutzerbedürfnisse. Sie verbindet damit ,, Distributed Data Processing" als ein technologisches Konzept mit ..Benutzerorientierung". Die Gestaltungsbereiche der Benutzerorientierung - Arbeitsmittel und Arbeitsumwelt, Mensch- Computer-Interaktionsschnittstelle sowie die Arbeitsorganisation - werden erläutert. Gestaltungsmaßnahmen werden beispielhaft angegeben, und es wird auf die weiterführende Literatur verwiesen; dabei steht das im Oldenbourg- Verlag erschienene Buch ,,Computerleistung am Arbeitsplatz - benutzerorientiertes Distributed Data Processing" im Vordergrund.

Sign in / Sign up

Export Citation Format

Share Document