scholarly journals A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

2022 ◽  
Vol 6 (1) ◽  
pp. 5
Author(s):  
Giuseppe Di Modica ◽  
Orazio Tomarchio

In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

2017 ◽  
Vol 2017 ◽  
pp. 1-11 ◽  
Author(s):  
Mukhtaj Khan ◽  
Zhengwen Huang ◽  
Maozhen Li ◽  
Gareth A. Taylor ◽  
Phillip M. Ashton ◽  
...  

The rapid deployment of Phasor Measurement Units (PMUs) in power systems globally is leading to Big Data challenges. New high performance computing techniques are now required to process an ever increasing volume of data from PMUs. To that extent the Hadoop framework, an open source implementation of the MapReduce computing model, is gaining momentum for Big Data analytics in smart grid applications. However, Hadoop has over 190 configuration parameters, which can have a significant impact on the performance of the Hadoop framework. This paper presents an Enhanced Parallel Detrended Fluctuation Analysis (EPDFA) algorithm for scalable analytics on massive volumes of PMU data. The novel EPDFA algorithm builds on an enhanced Hadoop platform whose configuration parameters are optimized by Gene Expression Programming. Experimental results show that the EPDFA is 29 times faster than the sequential DFA in processing PMU data and 1.87 times faster than a parallel DFA, which utilizes the default Hadoop configuration settings.


2020 ◽  
Vol 12 (7) ◽  
pp. 113 ◽  
Author(s):  
Maurizio Capra ◽  
Beatrice Bussolino ◽  
Alberto Marchisio ◽  
Muhammad Shafique ◽  
Guido Masera ◽  
...  

Deep Neural Networks (DNNs) are nowadays a common practice in most of the Artificial Intelligence (AI) applications. Their ability to go beyond human precision has made these networks a milestone in the history of AI. However, while on the one hand they present cutting edge performance, on the other hand they require enormous computing power. For this reason, numerous optimization techniques at the hardware and software level, and specialized architectures, have been developed to process these models with high performance and power/energy efficiency without affecting their accuracy. In the past, multiple surveys have been reported to provide an overview of different architectures and optimization techniques for efficient execution of Deep Learning (DL) algorithms. This work aims at providing an up-to-date survey, especially covering the prominent works from the last 3 years of the hardware architectures research for DNNs. In this paper, the reader will first understand what a hardware accelerator is, and what are its main components, followed by the latest techniques in the field of dataflow, reconfigurability, variable bit-width, and sparsity.


Author(s):  
A. K. Tripathi ◽  
S. Agrawal ◽  
R. D. Gupta

<p><strong>Abstract.</strong> The emergence of new tools and technologies to gather the information generate the problem of processing spatial big data. The solution of this problem requires new research, techniques, innovation and development. Spatial big data is categorized by the five V’s: volume, velocity, veracity, variety and value. Hadoop is a most widely used framework which address these problems. But it requires high performance computing resources to store and process such huge data. The emergence of cloud computing has provided, on demand, elastic, scalable and payment based computing resources to users to develop their own computing environment. The main objective of this paper is to develop a cloud enabled hadoop framework which combines cloud technology and high computing resources with the conventional hadoop framework to support the spatial big data solutions. The paper also compares the conventional hadoop framework and proposed cloud enabled hadoop framework. It is observed that the propose cloud enabled hadoop framework is much efficient to spatial big data processing than the current available solutions.</p>


Author(s):  
Samson Aregawi ◽  
Abiy Goshu ◽  
Bisrat Alemu ◽  
Dagmay Woldaregay ◽  
Nathnael Abdulkadir ◽  
...  

The concept of microbial concrete is one of the recent advances in concrete technology. In the past two decades, concrete technologies are working towards developing high performance concrete. Researches over the globe are being carried out in the wake of promising results found on the improvement of cementitious mix performances due to the application of live microorganisms. In this research live microorganism named Sporosarcina pasteurii, soil bacterium, has been used. Different set of experiments were carried out to investigate the effect of the bacterial medium, bacterial nutrient and bacterial concentration. From the test results it was found out that the bacterial medium had little effect, while the bacterial nutrient, whose main constituent is yeast extract, significantly reduced the compressive strength and increased the flow table as well as the slump in both mortar and concrete. Those samples with aforementioned bacteria together with the bacterial nutrient showed an improved compressive strength. The micro behaviors observed in terms of compressive strength indicate that this gain of strength was due to the calcite precipitation induced by the bacteria. The paper concludes by stating the need of further investigation, especially with regards to finding a better substitute for yeast extract of the bacterial nutrient.


2020 ◽  
Vol 13 (4) ◽  
pp. 790-797
Author(s):  
Gurjit Singh Bhathal ◽  
Amardeep Singh Dhiman

Background: In current scenario of internet, large amounts of data are generated and processed. Hadoop framework is widely used to store and process big data in a highly distributed manner. It is argued that Hadoop Framework is not mature enough to deal with the current cyberattacks on the data. Objective: The main objective of the proposed work is to provide a complete security approach comprising of authorisation and authentication for the user and the Hadoop cluster nodes and to secure the data at rest as well as in transit. Methods: The proposed algorithm uses Kerberos network authentication protocol for authorisation and authentication and to validate the users and the cluster nodes. The Ciphertext-Policy Attribute- Based Encryption (CP-ABE) is used for data at rest and data in transit. User encrypts the file with their own set of attributes and stores on Hadoop Distributed File System. Only intended users can decrypt that file with matching parameters. Results: The proposed algorithm was implemented with data sets of different sizes. The data was processed with and without encryption. The results show little difference in processing time. The performance was affected in range of 0.8% to 3.1%, which includes impact of other factors also, like system configuration, the number of parallel jobs running and virtual environment. Conclusion: The solutions available for handling the big data security problems faced in Hadoop framework are inefficient or incomplete. A complete security framework is proposed for Hadoop Environment. The solution is experimentally proven to have little effect on the performance of the system for datasets of different sizes.


No other talent process has been the subject of such great debate and emotion as performance management (PM). For decades, different strategies have been tried to improve PM processes, yielding an endless cycle of reform to capture the next “flavor-of-the-day” PM trend. The past 5 years, however, have brought novel thinking that is different from past trends. Companies are reducing their formal processes, driving performance-based cultures, and embedding effective PM behavior into daily work rather than relying on annual reviews to drive these. Through case studies provided from leading organizations, this book illustrates the range of PM processes that companies are using today. These show a shift away from adopting someone else’s best practice; instead, companies are designing bespoke PM processes that fit their specific strategy, climate, and needs. Leading PM thought leaders offer their views about the state of PM today, what we have learned and where we need to focus future efforts, including provocative new research that shows what matters most in driving high performance. This book is a call to action for talent management professionals to go beyond traditional best practice and provide thought leadership in designing PM processes and systems that will enhance both individual and organizational performance.


Author(s):  
Djordje Romanic

Tornadoes and downbursts cause extreme wind speeds that often present a threat to human safety, structures, and the environment. While the accuracy of weather forecasts has increased manifold over the past several decades, the current numerical weather prediction models are still not capable of explicitly resolving tornadoes and small-scale downbursts in their operational applications. This chapter describes some of the physical (e.g., tornadogenesis and downburst formation), mathematical (e.g., chaos theory), and computational (e.g., grid resolution) challenges that meteorologists currently face in tornado and downburst forecasting.


Author(s):  
Liping Yao ◽  
Danlei Zhu ◽  
Hailiang Liao ◽  
Sheik Haseena ◽  
Mahesh kumar Ravva ◽  
...  

Due to their advantages of low-cost, light-weight, and mechanical flexibility, much attention has been focused on pi-conjugated organic semiconductors. In the past decade, although many materials with high performance has...


Sign in / Sign up

Export Citation Format

Share Document