Various Approaches Proposed for Eliminating Duplicate Data in a System

The growth of big data processing market led to an increase in the overload of computation data centers, change of methods used in storing the data, communication between the computing units and computational time needed to process or edit the data. Methods of distributed or parallel data processing brought new problems related to computations with data which need to be examined. Unlike the conventional cloud services, a tight connection between the data and the computations is one of the main characteristics of the big data services. The computational tasks can be done only if relevant data are available. Three factors, which influence the speed and efficiency of data processing are - data duplicity, data integrity and data security. We are motivated to study the problems related to the growing time needed for data processing by optimizing these three factors in geographically distributed data centers.

Download Full-text

Cost Minimization for Big Data Processing in Geo-Distributed Data Centers

IEEE Transactions on Emerging Topics in Computing ◽

10.1109/tetc.2014.2310456 ◽

2014 ◽

Vol 2 (3) ◽

pp. 314-323 ◽

Cited By ~ 63

Author(s):

Lin Gu ◽

Deze Zeng ◽

Peng Li ◽

Song Guo

Keyword(s):

Big Data ◽

Data Processing ◽

Data Centers ◽

Cost Minimization ◽

Distributed Data ◽

Big Data Processing

Download Full-text

Cost Minimization for Big Data Processing in Geo-Distributed Data Centers

Wireless Networks - Cloud Networking for Big Data ◽

10.1007/978-3-319-24720-5_4 ◽

2015 ◽

pp. 59-78 ◽

Cited By ~ 6

Author(s):

Deze Zeng ◽

Lin Gu ◽

Song Guo

Keyword(s):

Big Data ◽

Data Processing ◽

Data Centers ◽

Cost Minimization ◽

Distributed Data ◽

Big Data Processing

Download Full-text

GreenCloudNet++: Simulation framework for energy efficient and secure, green job scheduling in geographically distributed data centers

Transactions on Emerging Telecommunications Technologies ◽

10.1002/ett.4232 ◽

2021 ◽

Author(s):

Farrukh Mahmood ◽

Farrukh Zeeshan Khan ◽

Muneer Ahmed ◽

Iftikhar Ahmad ◽

Brij B. Gupta

Keyword(s):

Energy Efficient ◽

Data Centers ◽

Job Scheduling ◽

Distributed Data ◽

Simulation Framework ◽

Geographically Distributed

Download Full-text

Sparkmach: A Distributed Data Processing System Based on Automated Machine Learning for Big Data

Information Management and Big Data - Communications in Computer and Information Science ◽

10.1007/978-3-030-11680-4_13 ◽

2019 ◽

pp. 121-128

Author(s):

Gusseppe Bravo-Rocca ◽

Piero Torres-Robatty ◽

Jose Fiestas-Iquira

Keyword(s):

Machine Learning ◽

Big Data ◽

Data Processing ◽

Processing System ◽

Distributed Data ◽

Data Processing System ◽

Distributed Data Processing ◽

Automated Machine Learning

Download Full-text

Optimization-based workload distribution in geographically distributed data centers: A survey

International Journal of Communication Systems ◽

10.1002/dac.4453 ◽

2020 ◽

Vol 33 (12) ◽

pp. e4453 ◽

Cited By ~ 3

Author(s):

Iftikhar Ahmad ◽

Muhammad Imran Khan Khalil ◽

Syed Adeel Ali Shah

Keyword(s):

Data Centers ◽

Distributed Data ◽

Geographically Distributed ◽

Workload Distribution

Download Full-text

The Berlin Big Data Center (BBDC)

it - Information Technology ◽

10.1515/itit-2018-0016 ◽

2018 ◽

Vol 60 (5-6) ◽

pp. 321-326 ◽

Cited By ~ 1

Author(s):

Christoph Boden ◽

Tilmann Rabl ◽

Volker Markl

Keyword(s):

Big Data ◽

Data Analysis ◽

Data Processing ◽

Deep Understanding ◽

Automatic Parallelization ◽

Second Phase ◽

Distributed Data ◽

Domain Specific ◽

Distributed Data Processing ◽

Large Groups

Abstract The last decade has been characterized by the collection and availability of unprecedented amounts of data due to rapidly decreasing storage costs and the omnipresence of sensors and data-producing global online-services. In order to process and analyze this data deluge, novel distributed data processing systems resting on the paradigm of data flow such as Apache Hadoop, Apache Spark, or Apache Flink were built and have been scaled to tens of thousands of machines. However, writing efficient implementations of data analysis programs on these systems requires a deep understanding of systems programming, prohibiting large groups of data scientists and analysts from efficiently using this technology. In this article, we present some of the main achievements of the research carried out by the Berlin Big Data Cente (BBDC). We introduce the two domain-specific languages Emma and LARA, which are deeply embedded in Scala and enable declarative specification and the automatic parallelization of data analysis programs, the PEEL Framework for transparent and reproducible benchmark experiments of distributed data processing systems, approaches to foster the interpretability of machine learning models and finally provide an overview of the challenges to be addressed in the second phase of the BBDC.

Download Full-text

The cloud of geographically distributed data centers

EPJ Web of Conferences ◽

10.1051/epjconf/201921407007 ◽

2019 ◽

Vol 214 ◽

pp. 07007

Author(s):

Petr Fedchenkov ◽

Andrey Shevel ◽

Sergey Khoruzhnikov ◽

Oleg Sadov ◽

Oleg Lazo ◽

...

Keyword(s):

Data Storage ◽

Data Centers ◽

Virtual Machines ◽

Data Transfer ◽

Service Availability ◽

Distributed Data ◽

Virtual Storage ◽

Distribution Services ◽

Geographically Distributed ◽

Data Centres

ITMO University (ifmo.ru) is developing the cloud of geographically distributed data centres. The geographically distributed means data centres (DC) located in different places far from each other by hundreds or thousands of kilometres. Usage of the geographically distributed data centres promises a number of advantages for end users such as opportunity to add additional DC and service availability through redundancy and geographical distribution. Services like data transfer, computing, and data storage are provided to users in the form of virtual objects including virtual machines, virtual storage, virtual data transfer link.

Download Full-text