Software-Defined Networking for Scalable Cloud-based Services to Improve System Performance of Hadoop-based Big Data Applications

2016 ◽  
Vol 8 (2) ◽  
pp. 1-22 ◽  
Author(s):  
Desta Haileselassie Hagos
Keyword(s):  
Big Data ◽  
System Performance ◽  
Dynamic Network ◽  
Software Defined Networking ◽  
Network Control ◽  
Virtual Networks ◽  
Big Data Applications ◽  
New Research ◽  
Cloud Infrastructures ◽  
The Impact

The rapid growth of Cloud Computing has brought with it major new challenges in the automated manageability, dynamic network reconfiguration, provisioning, scalability and flexibility of virtual networks. OpenFlow-enabled Software-Defined Networking (SDN) alleviates these key challenges through the abstraction of lower level functionality that removes the complexities of the underlying hardware by separating the data and control planes. SDN has an efficient, dynamic, automated network management, higher availability and application provisioning through programmable interfaces which are very critical for flexible and scalable cloud-based services. In this study, the author explores broadly useful open technologies and methodologies for applying an OpenFlow-enabled SDN to scalable cloud-based services and a variety of diverse applications. The approach in this paper introduces new research challenges in the design and implementation of advanced techniques for bringing an SDN-enabled components and big data applications into a cloud environment in a dynamic setting. Some of these challenges become pressing concerns to cloud providers when managing virtual networks and data centers, while others complicate the development and deployment of cloud-hosted applications from the perspective of developers and end users. However, the growing demand for manageable, scalable and flexible clouds necessitates that effective solutions to these challenges be found. Hence, through real-world research validation use cases, this paper aims at exploring useful mechanisms for the role and potential of an OpenFlow-enabled SDN and its direct benefit for scalable cloud-based services. Finally, it demonstrates the impact of an OpenFlow-enabled SDN that fully embraces the opportunities and challenges of cloud infrastructures to improve the system performance of Hadoop-based big data applications by utilizing the network control capabilities of an OpenFlow to solve network congestion.

Software-Defined Networking for Scalable Cloud-Based Services to Improve System Performance of Hadoop-Based Big Data Applications

Web Services ◽  
2019 ◽  
pp. 1460-1484
Author(s):  
Desta Haileselassie Hagos
Keyword(s):  
Big Data ◽  
System Performance ◽  
Dynamic Network ◽  
Software Defined Networking ◽  
Network Control ◽  
Virtual Networks ◽  
Big Data Applications ◽  
New Research ◽  
Cloud Infrastructures ◽  
And Control

The rapid growth of Cloud Computing has brought with it major new challenges in the automated manageability, dynamic network reconfiguration, provisioning, scalability and flexibility of virtual networks. OpenFlow-enabled Software-Defined Networking (SDN) alleviates these key challenges through the abstraction of lower level functionality that removes the complexities of the underlying hardware by separating the data and control planes. SDN has an efficient, dynamic, automated network management, higher availability and application provisioning through programmable interfaces which are very critical for flexible and scalable cloud-based services. In this study, the author explores broadly useful open technologies and methodologies for applying an OpenFlow-enabled SDN to scalable cloud-based services and a variety of diverse applications. The approach in this paper introduces new research challenges in the design and implementation of advanced techniques for bringing an SDN-enabled components and big data applications into a cloud environment in a dynamic setting. Some of these challenges become pressing concerns to cloud providers when managing virtual networks and data centers, while others complicate the development and deployment of cloud-hosted applications from the perspective of developers and end users. However, the growing demand for manageable, scalable and flexible clouds necessitates that effective solutions to these challenges be found. Hence, through real-world research validation use cases, this paper aims at exploring useful mechanisms for the role and potential of an OpenFlow-enabled SDN and its direct benefit for scalable cloud-based services. Finally, it demonstrates the impact of an OpenFlow-enabled SDN that fully embraces the opportunities and challenges of cloud infrastructures to improve the system performance of Hadoop-based big data applications by utilizing the network control capabilities of an OpenFlow to solve network congestion.

scholarly journals Power Budgeting of Big Data Applications in Container-based Clusters

2020 ◽  
Author(s):  
Jonatan Enes ◽  
Guillaume Fieni ◽  
Roberto R. Exposito ◽  
Romain Rouvoy ◽  
Juan Tourino
Keyword(s):  
Big Data ◽  
Big Data Applications

scholarly journals Computational storage: an efficient and scalable platform for big data and HPC applications

2019 ◽  
Vol 6 (1) ◽  
Author(s):  
Mahdi Torabzadehkashi ◽  
Siavash Rezaei ◽  
Ali HeydariGorji ◽  
Hosein Bobarshad ◽  
Vladimir Alves ◽  
...  
Keyword(s):  
Big Data ◽  
High Performance ◽  
Distributed Processing ◽  
Data Access ◽  
Distributed Applications ◽  
Process Data ◽  
Storage Devices ◽  
Hadoop Mapreduce ◽  
Big Data Applications ◽  
Application Processor

AbstractIn the era of big data applications, the demand for more sophisticated data centers and high-performance data processing mechanisms is increasing drastically. Data are originally stored in storage systems. To process data, application servers need to fetch them from storage devices, which imposes the cost of moving data to the system. This cost has a direct relation with the distance of processing engines from the data. This is the key motivation for the emergence of distributed processing platforms such as Hadoop, which move process closer to data. Computational storage devices (CSDs) push the “move process to data” paradigm to its ultimate boundaries by deploying embedded processing engines inside storage devices to process data. In this paper, we introduce Catalina, an efficient and flexible computational storage platform, that provides a seamless environment to process data in-place. Catalina is the first CSD equipped with a dedicated application processor running a full-fledged operating system that provides filesystem-level data access for the applications. Thus, a vast spectrum of applications can be ported for running on Catalina CSDs. Due to these unique features, to the best of our knowledge, Catalina CSD is the only in-storage processing platform that can be seamlessly deployed in clusters to run distributed applications such as Hadoop MapReduce and HPC applications in-place without any modifications on the underlying distributed processing framework. For the proof of concept, we build a fully functional Catalina prototype and a CSD-equipped platform using 16 Catalina CSDs to run Intel HiBench Hadoop and HPC benchmarks to investigate the benefits of deploying Catalina CSDs in the distributed processing environments. The experimental results show up to 2.2× improvement in performance and 4.3× reduction in energy consumption, respectively, for running Hadoop MapReduce benchmarks. Additionally, thanks to the Neon SIMD engines, the performance and energy efficiency of DFT algorithms are improved up to 5.4× and 8.9×, respectively.

Preliminary Benefits of Big Data in the Construction Industry: A Case Study

2022 ◽  
pp. 1-11
Author(s):  
Bernard Tuffour Atuahene ◽  
Sittimont Kanjanabootra ◽  
Thayaparan Gajendran
Keyword(s):  
Big Data ◽  
Construction Industry ◽  
Construction Projects ◽  
Big Data Applications ◽  
Data Application ◽  
Construction Firm ◽  
Big Data Application ◽  
Tangible Benefit ◽  
Design Construction

Big data applications consist of i) data collection using big data sources, ii) storing and processing the data, and iii) analysing data to gain insights for creating organisational benefit. The influx of digital technologies and digitization in the construction process includes big data as one newly emerging digital technology adopted in the construction industry. Big data application is in a nascent stage in construction, and there is a need to understand the tangible benefit(s) that big data can offer the construction industry. This study explores the benefits of big data in the construction industry. Using a qualitative case study design, construction professionals in an Australian Construction firm were interviewed. The research highlights that the benefits of big data include reduction of litigation amongst projects stakeholders, enablement of near to real-time communication, and facilitation of effective subcontractor selection. By implication, on a broader scale, these benefits can improve contract management, procurement, and management of construction projects. This study contributes to an ongoing discourse on big data application, and more generally, digitization in the construction industry.

Sign in / Sign up

Export Citation Format

Share Document