Scalable Data Analytics from Predevelopment to Large Scale Manufacturing

The Engineer Research and Development Center, Information Technology Laboratory’s (ERDC-ITL’s) Big Data Analytics team specializes in the analysis of large-scale datasets with capabilities across four research areas that require vast amounts of data to inform and drive analysis: large-scale data governance, deep learning and machine learning, natural language processing, and automated data labeling. Unfortunately, data transfer between government organizations is a complex and time-consuming process requiring coordination of multiple parties across multiple offices and organizations. Past successes in large-scale data analytics have placed a significant demand on ERDC-ITL researchers, highlighting that few individuals fully understand how to successfully transfer data between government organizations; future project success therefore depends on a small group of individuals to efficiently execute a complicated process. The Big Data Analytics team set out to develop a standardized workflow for the transfer of large-scale datasets to ERDC-ITL, in part to educate peers and future collaborators on the process required to transfer datasets between government organizations. Researchers also aim to increase workflow efficiency while protecting data integrity. This report provides an overview of the created Data Lake Ecosystem Workflow by focusing on the six phases required to efficiently transfer large datasets to supercomputing resources located at ERDC-ITL.

Download Full-text

Software Abstractions for Large-Scale Deep Learning Models in Big Data Analytics

International Journal of Advanced Computer Science and Applications ◽

10.14569/ijacsa.2019.0100469 ◽

2019 ◽

Vol 10 (4) ◽

Author(s):

Ayaz H Khan ◽

Ali Mustafa ◽

Aneeq Yusuf ◽

Rehanullah Khan

Keyword(s):

Big Data ◽

Deep Learning ◽

Data Analytics ◽

Large Scale ◽

Big Data Analytics ◽

Learning Models

Download Full-text

On-demand self-adaptive data analytics in large-scale decentralized networks

2017 IEEE 16th International Symposium on Network Computing and Applications (NCA) ◽

10.1109/nca.2017.8171353 ◽

2017 ◽

Cited By ~ 2

Author(s):

Evangelos Pournaras ◽

Jovan Nikolic

Keyword(s):

Data Analytics ◽

Large Scale ◽

Decentralized Networks ◽

On Demand ◽

Self Adaptive

Download Full-text

The Intersection of Data Analytics and Data-Driven Innovation

AI and Big Data’s Potential for Disruptive Innovation - Advances in Computational Intelligence and Robotics ◽

10.4018/978-1-5225-9687-5.ch012 ◽

2020 ◽

pp. 317-343

Author(s):

Marcus Tanque ◽

Harry J. Foxwell

Keyword(s):

Private Sector ◽

Data Analytics ◽

Large Scale ◽

Data Driven ◽

Public And Private ◽

Managerial Practices ◽

Project Completion ◽

Technological Advances ◽

Private Sector Organizations ◽

Public And Private Sector

This chapter discusses businesses, key technology implementations, case studies, limitations, and trends. It also presents recommendations to improve data analysis, data-driven innovation, and big data project implementation. Small-to-large-scale project inefficiencies present unique challenges to both public and private sector institutions and their management. Data analytics management, data-driven innovation, and related project initiatives have grown in scope, scale, and frequency. This evolution is due to continued technological advances in analytical methods and computing technologies. Most public and private sector organizations do not deliver on project benefits and results. Many organizational and managerial practices emphasize these technical limitations. Specialized human and technical resources are essential for an organization's effective project completion. Functional and practical areas affecting analytics domain and ability requirements, stakeholder expectations, solution infrastructure choices, legal and ethical concerns will also be discussed in this chapter.

Download Full-text

Big Data and Cloud Computing

Exploring the Convergence of Big Data and the Internet of Things - Advances in Data Mining and Database Management ◽

10.4018/978-1-5225-2947-7.ch001 ◽

2018 ◽

pp. 1-28 ◽

Cited By ~ 2

Author(s):

Marcus Tanque ◽

Harry J Foxwell

Keyword(s):

Cloud Computing ◽

Big Data ◽

Supply Chain ◽

Supply Chain Management ◽

Data Analytics ◽

Large Scale ◽

Big Data Analytics ◽

Chain Management ◽

Computing Services ◽

Cloud Computing Services

Big data and cloud computing are transforming information technology. These comparable technologies are the result of dramatic developments in computational power, virtualization, network bandwidth, availability, storage capability, and cyber-physical systems. The crossroads of these two areas, involves the use of cloud computing services and infrastructure, to support large-scale data analytics research, providing relevant solutions or future possibilities for supply chain management. This chapter broadens the current posture of cloud computing and big data, as associate with the supply chain solutions. This chapter focuses on areas of significant technology and scientific advancements, which are likely to enhance supply chain systems. This evaluation emphasizes the security challenges and mega-trends affecting cloud computing and big data analytics pertaining to supply chain management.

Download Full-text

Synchronizing Execution of Big Data in Distributed and Parallelized Environments

Big Data ◽

10.4018/978-1-4666-9840-6.ch071 ◽

2016 ◽

pp. 1555-1581

Author(s):

Gueyoung Jung ◽

Tridib Mukherjee

Keyword(s):

Big Data ◽

Distributed System ◽

Data Analytics ◽

High Performance ◽

Large Scale ◽

Big Data Analytics ◽

Loosely Coupled ◽

Current Trends ◽

Distributed Computing Infrastructures ◽

Performance Computing

In the modern information era, the amount of data has exploded. Current trends further indicate exponential growth of data in the future. This prevalent humungous amount of data—referred to as big data—has given rise to the problem of finding the “needle in the haystack” (i.e., extracting meaningful information from big data). Many researchers and practitioners are focusing on big data analytics to address the problem. One of the major issues in this regard is the computation requirement of big data analytics. In recent years, the proliferation of many loosely coupled distributed computing infrastructures (e.g., modern public, private, and hybrid clouds, high performance computing clusters, and grids) have enabled high computing capability to be offered for large-scale computation. This has allowed the execution of the big data analytics to gather pace in recent years across organizations and enterprises. However, even with the high computing capability, it is a big challenge to efficiently extract valuable information from vast astronomical data. Hence, we require unforeseen scalability of performance to deal with the execution of big data analytics. A big question in this regard is how to maximally leverage the high computing capabilities from the aforementioned loosely coupled distributed infrastructure to ensure fast and accurate execution of big data analytics. In this regard, this chapter focuses on synchronous parallelization of big data analytics over a distributed system environment to optimize performance.

Download Full-text

Research Challenges in Big Data Analytics

Decision Management ◽

10.4018/978-1-5225-1837-2.ch006 ◽

2017 ◽

pp. 83-99

Author(s):

Sivamathi Chokkalingam ◽

Vijayarani S.

Keyword(s):

Big Data ◽

Data Analytics ◽

Large Scale ◽

New Technologies ◽

Big Data Analytics ◽

Large Data ◽

Data Sets ◽

Data Types ◽

Customer Preferences ◽

Research Challenges

The term Big Data refers to large-scale information management and analysis technologies that exceed the capability of traditional data processing technologies. Big Data is differentiated from traditional technologies in three ways: volume, velocity and variety of data. Big data analytics is the process of analyzing large data sets which contains a variety of data types to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information. Since Big Data is new emerging field, there is a need for development of new technologies and algorithms for handling big data. The main objective of this paper is to provide knowledge about various research challenges of Big Data analytics. A brief overview of various types of Big Data analytics is discussed in this paper. For each analytics, the paper describes process steps and tools. A banking application is given for each analytics. Some of research challenges and possible solutions for those challenges of big data analytics are also discussed.

Download Full-text