scholarly journals Cost-Effective Resource Provisioning for Real-Time Workflow in Cloud

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-15
Author(s):  
Lei Wu ◽  
Ran Ding ◽  
Zhaohong Jia ◽  
Xuejun Li
Keyword(s):  
Big Data ◽  
Data Management ◽  
Real Time ◽  
Critical Path ◽  
Cost Effective ◽  
Resource Provisioning ◽  
Complex Data ◽  
Volume Data ◽  
Workflow Execution ◽  
Time Requirements

In the era of big data, mining and analysis of the enormous amount of data has been widely used to support decision-making. This complex process including huge-volume data collecting, storage, transmission, and analysis could be modeled as workflow. Meanwhile, cloud environment provides sufficient computing and storage resources for big data management and analytics. Due to the clouds providing the pay-as-you-go pricing scheme, executing a workflow in clouds should pay for the provisioned resources. Thus, cost-effective resource provisioning for workflow in clouds is still a critical challenge. Also, the responses of the complex data management process are usually required to be real-time. Therefore, deadline is the most crucial constraint for workflow execution. In order to address the challenge of cost-effective resource provisioning while meeting the real-time requirements of workflow execution, a resource provisioning strategy based on dynamic programming is proposed to achieve cost-effectiveness of workflow execution in clouds and a critical-path based workflow partition algorithm is presented to guarantee that the workflow can be completed before deadline. Our approach is evaluated by simulation experiments with real-time workflows of different sizes and different structures. The results demonstrate that our algorithm outperforms the existing classical algorithms.

Our Knowledge Management Hubble May Need Glasses

2018 ◽  
Vol 14 (1) ◽  
pp. 30-50 ◽  
Author(s):  
William H. Money ◽  
Stephen J. Cohen
Keyword(s):  
Big Data ◽  
Knowledge Management ◽  
Real Time ◽  
Complex Data ◽  
Data Systems ◽  
Fault Handling ◽  
Architectural Framework ◽  
Big Data Systems ◽  
And Performance ◽  
Streaming Big Data

This article analyzes the properties of unknown faults in knowledge management and Big Data systems processing Big Data in real-time. These faults introduce risks and threaten the knowledge pyramid and decisions based on knowledge gleaned from volumes of complex data. The authors hypothesize that not yet encountered faults may require fault handling, an analytic model, and an architectural framework to assess and manage the faults and mitigate the risks of correlating or integrating otherwise uncorrelated Big Data, and to ensure the source pedigree, quality, set integrity, freshness, and validity of the data. New architectures, methods, and tools for handling and analyzing Big Data systems functioning in real-time will contribute to organizational knowledge and performance. System designs must mitigate faults resulting from real-time streaming processes while ensuring that variables such as synchronization, redundancy, and latency are addressed. This article concludes that with improved designs, real-time Big Data systems may continuously deliver the value of streaming Big Data.

A Generation Algorithm of DRR Based on GPU

2014 ◽  
Vol 543-547 ◽  
pp. 2809-2812
Author(s):  
Xiao Gang Du ◽  
Jian Wu Dang ◽  
Yang Ping Wang
Keyword(s):  
Real Time ◽  
X Rays ◽  
Volume Data ◽  
Generation Algorithm ◽  
Image Guided Radiotherapy ◽  
Device Architecture ◽  
Digitally Reconstructed Radiographs ◽  
Time Requirements ◽  
Generation Procedure

Because of the good parallelism of the generation procedure of digitally reconstructed radiographs, a real-time generation algorithm of digitally reconstructed radiographs based on Compute Unified Device Architecture is presented in this paper. Firstly, the volume data and other input parameters are read and loaded into the GPU; Secondly, the kernel procedure which can be used to simulate the decay process of X-ray in the human body is designed according to the correspondence between X-rays and threads; Finally, the kernel function is executed in parallel by the multi-thread to complete the DRR image generation. The experimental results show that this algorithm uses effectively the parallel computing capabilities of GPU in the premise of ensuring the quality of the DRR, improves significantly the generation speed of DRR, and meets the real-time requirements of digitally reconstructed radiographs in the image-guided radiotherapy.

scholarly journals Dynamic Fault-Tolerant Workflow Scheduling with Hybrid Spatial-Temporal Re-Execution in Clouds

Information ◽  
2019 ◽  
Vol 10 (5) ◽  
pp. 169 ◽  
Author(s):  
Na Wu ◽  
Decheng Zuo ◽  
Zhan Zhang
Keyword(s):  
Large Scale ◽  
Fault Tolerant ◽  
Critical Path ◽  
High Reliability ◽  
Low Cost ◽  
Cost Effective ◽  
Scientific Workflow ◽  
Workflow Scheduling ◽  
Workflow Execution ◽  
Computing Environments

Improving reliability is one of the major concerns of scientific workflow scheduling in clouds. The ever-growing computational complexity and data size of workflows present challenges to fault-tolerant workflow scheduling. Therefore, it is essential to design a cost-effective fault-tolerant scheduling approach for large-scale workflows. In this paper, we propose a dynamic fault-tolerant workflow scheduling (DFTWS) approach with hybrid spatial and temporal re-execution schemes. First, DFTWS calculates the time attributes of tasks and identifies the critical path of workflow in advance. Then, DFTWS assigns appropriate virtual machine (VM) for each task according to the task urgency and budget quota in the phase of initial resource allocation. Finally, DFTWS performs online scheduling, which makes real-time fault-tolerant decisions based on failure type and task criticality throughout workflow execution. The proposed algorithm is evaluated on real-world workflows. Furthermore, the factors that affect the performance of DFTWS are analyzed. The experimental results demonstrate that DFTWS achieves a trade-off between high reliability and low cost objectives in cloud computing environments.

ZAMREN Big Data Management (ZAMBiDM) Envisaging Efficiency and Analytically Manage IT Resources

Web Services ◽  
2019 ◽  
pp. 1802-1811
Author(s):  
Jameson Mbale
Keyword(s):  
Decision Making ◽  
Big Data ◽  
Data Management ◽  
Real Time ◽  
Economic Gain ◽  
Teaching And Research ◽  
Strategic Actions ◽  
Huge Data ◽  
It Resources

The ZAMREN member institutions deal with heterogeneous teaching and research materials drawn from all walks-of-life such as industry, and NRENs world over. To deal with such huge data that is in terabits for academic and economic gain becomes a mammoth task to manipulate, process, store and analyse. It is in view of that the ZAMREN Big Data and Data Management, in this work abbreviated as ZAMBiDM, is envisaged to collectively gather relevant heterogeneous large volumes of a wide variety of data from all sectors of economy. The data would be analytically managed in storage, processing and obtaining actionable insight real-time as a way to solve high-value skilled academic and industrial business problems, in order to prepare graduates for competitive future workforce. The data would be collected from all line-ministries of Zambia such as education, agriculture, health, mining, lands, communications, commerce, including industries and NRENs worldwide and be analytically analysed to exploit strategic actions that would enhance decision making in executing relevant tasks.

ZAMREN Big Data Management (ZAMBiDM) Envisaging Efficiency and Analytically Manage IT Resources

2016 ◽  
pp. 55-67
Author(s):  
Jameson Mbale
Keyword(s):  
Decision Making ◽  
Big Data ◽  
Data Management ◽  
Real Time ◽  
Economic Gain ◽  
Teaching And Research ◽  
Strategic Actions ◽  
Huge Data ◽  
It Resources

The ZAMREN member institutions deal with heterogeneous teaching and research materials drawn from all walks-of-life such as industry, and NRENs world over. To deal with such huge data that is in terabits for academic and economic gain becomes a mammoth task to manipulate, process, store and analyse. It is in view of that the ZAMREN Big Data and Data Management, in this work abbreviated as ZAMBiDM, is envisaged to collectively gather relevant heterogeneous large volumes of a wide variety of data from all sectors of economy. The data would be analytically managed in storage, processing and obtaining actionable insight real-time as a way to solve high-value skilled academic and industrial business problems, in order to prepare graduates for competitive future workforce. The data would be collected from all line-ministries of Zambia such as education, agriculture, health, mining, lands, communications, commerce, including industries and NRENs worldwide and be analytically analysed to exploit strategic actions that would enhance decision making in executing relevant tasks.

scholarly journals Requirements of Health Data Management Systems for Biomedical Care and Research: Scoping Review (Preprint)

2019 ◽  
Author(s):  
Leila Ismail ◽  
Huned Materwala ◽  
Achim P Karduck ◽  
Abdu Adem
Keyword(s):  
Big Data ◽  
Data Management ◽  
Real Time ◽  
Management System ◽  
Data Analytics ◽  
Big Data Analytics ◽  
Health Data ◽  
Data Management System ◽  
Management Systems ◽  
Data Management Systems

BACKGROUND Over the last century, disruptive incidents in the fields of clinical and biomedical research have yielded a tremendous change in health data management systems. This is due to a number of breakthroughs in the medical field and the need for big data analytics and the Internet of Things (IoT) to be incorporated in a real-time smart health information management system. In addition, the requirements of patient care have evolved over time, allowing for more accurate prognoses and diagnoses. In this paper, we discuss the temporal evolution of health data management systems and capture the requirements that led to the development of a given system over a certain period of time. Consequently, we provide insights into those systems and give suggestions and research directions on how they can be improved for a better health care system. OBJECTIVE This study aimed to show that there is a need for a secure and efficient health data management system that will allow physicians and patients to update decentralized medical records and to analyze the medical data for supporting more precise diagnoses, prognoses, and public insights. Limitations of existing health data management systems were analyzed. METHODS To study the evolution and requirements of health data management systems over the years, a search was conducted to obtain research articles and information on medical lawsuits, health regulations, and acts. These materials were obtained from the Institute of Electrical and Electronics Engineers, the Association for Computing Machinery, Elsevier, MEDLINE, PubMed, Scopus, and Web of Science databases. RESULTS Health data management systems have undergone a disruptive transformation over the years from paper to computer, web, cloud, IoT, big data analytics, and finally to blockchain. The requirements of a health data management system revealed from the evolving definitions of medical records and their management are (1) medical record data, (2) real-time data access, (3) patient participation, (4) data sharing, (5) data security, (6) patient identity privacy, and (7) public insights. This paper reviewed health data management systems based on these 7 requirements across studies conducted over the years. To our knowledge, this is the first analysis of the temporal evolution of health data management systems giving insights into the system requirements for better health care. CONCLUSIONS There is a need for a comprehensive real-time health data management system that allows physicians, patients, and external users to input their medical and lifestyle data into the system. The incorporation of big data analytics will aid in better prognosis or diagnosis of the diseases and the prediction of diseases. The prediction results will help in the development of an effective prevention plan.

Design and Realization of a Distributed Big Data Management System

2014 ◽  
Vol 1030-1032 ◽  
pp. 1900-1904 ◽  
Author(s):  
Hai Yan Chen
Keyword(s):  
Big Data ◽  
Data Management ◽  
Real Time ◽  
Management System ◽  
Distributed Database ◽  
Data Management System ◽  
Processing Capacity ◽  
Relation Database ◽  
Mass Data ◽  
Permanent Storage

Big Data provides a possibility of handling mass data, which acts as a subversive technique. By the way, traditional relation database is no more effective of mass data that causes distributed database NoSQL to appear and evolve. In this article, we will design and realize a new distributed big data management system (DBDMS), which is based on Hadoop and NoSQL techniques, and it provides big data real-time collection, search and permanent storage. Proved by some experiment, DBDMS can enhance the processing capacity of mass data, very suitable for mass log backup and retrieval, mass network packet grab and analyze, and etc. other applied areas.

scholarly journals An Intelligent Approach for Handling Complexity by Migrating from Conventional Databases to Big Data

Symmetry ◽  
2018 ◽  
Vol 10 (12) ◽  
pp. 698 ◽  
Author(s):  
Shabana Ramzan ◽  
Imran Bajwa ◽  
Rafaqut Kazmi
Keyword(s):  
Big Data ◽  
Information Systems ◽  
Relational Databases ◽  
High Volume ◽  
Complex Data ◽  
Volume Data ◽  
Architectural Support ◽  
Time Interaction ◽  
Novel Approach ◽  
Intelligent Approach

Handling complexity in the data of information systems has emerged into a serious challenge in recent times. The typical relational databases have limited ability to manage the discrete and heterogenous nature of modern data. Additionally, the complexity of data in relational databases is so high that the efficient retrieval of information has become a bottleneck in traditional information systems. On the side, Big Data has emerged into a decent solution for heterogenous and complex data (structured, semi-structured and unstructured data) by providing architectural support to handle complex data and by providing a tool-kit for efficient analysis of complex data. For the organizations that are sticking to relational databases and are facing the challenge of handling complex data, they need to migrate their data to a Big Data solution to get benefits such as horizontal scalability, real-time interaction, handling high volume data, etc. However, such migration from relational databases to Big Data is in itself a challenge due to the complexity of data. In this paper, we introduce a novel approach that handles complexity of automatic transformation of existing relational database (MySQL) into a Big data solution (Oracle NoSQL). The used approach supports a bi-fold transformation (schema-to-schema and data-to-data) to minimize the complexity of data and to allow improved analysis of data. A software prototype for this transformation is also developed as a proof of concept. The results of the experiments show the correctness of our transformations that outperform the other similar approaches.

Sign in / Sign up

Export Citation Format

Share Document