Big data quality framework: a holistic approach to continuous quality management

2021 ◽  
Vol 8 (1) ◽  
Author(s):  
Ikbal Taleb ◽  
Mohamed Adel Serhani ◽  
Chafik Bouhaddioui ◽  
Rachida Dssouli
Keyword(s):  
Big Data ◽  
Quality Management ◽  
Data Quality ◽  
Value Added ◽  
Holistic Approach ◽  
Research Area ◽  
Heterogeneous Data ◽  
Data Generation ◽  
Continuous Quality ◽  
Quality Profile

AbstractBig Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.

scholarly journals Extracting and Transforming Heterogeneous Data from XML files for Big Data

2019 ◽  
Vol 9 (2) ◽  
pp. 4276-4280
Keyword(s):  
Machine Learning ◽  
Big Data ◽  
Learning Community ◽  
Digital Technology ◽  
Data Warehousing ◽  
Continuous Process ◽  
Heterogeneous Data ◽  
Data Generation ◽  
Data Systems ◽  
The Web

Digital technology is fast changing in the recent years and with this change, the number of data systems, sources, and formats has also increased exponentially. So the process of extracting data from these multiple source systems and transforming it to suit for various analytics processes is gaining importance at an alarming rate. In order to handle Big Data, the process of transformation is quite challenging, as data generation is a continuous process. In this paper, we extract data from various heterogeneous sources from the web and try to transform it into a form which is vastly used in data warehousing so that it caters to the analytical needs of the machine learning community.

scholarly journals Regional Development in the Age of Big Data

2018 ◽  
Vol 12 (1) ◽  
pp. 1-9
Author(s):  
József Jankó ◽  
György Szabó
Keyword(s):  
Big Data ◽  
Professional Practice ◽  
Spatial Planning ◽  
Landscape Architecture ◽  
Analytical Approach ◽  
Expert Knowledge ◽  
Value Added ◽  
Heterogeneous Data ◽  
Territorial Development ◽  
Related Data

Our paper presents a forward looking analytical approach to the territorial development in a region ofthe Transylvanian Plain situated in the vicinity of Cluj-Napoca, Romania. We outlined the development ofthis region with the means of landscape architecture supported by a comparable assessment. In the ageof Big Data we settled at creative usage of traditional analysis. We extracted yet undetected informationfrom a limited amount of available as yet loosely related data. The key feature of the employed model isthe ontological traceability of cause and effect. Although technology is available to collect enormous data,expert knowledge gained by education and professional practice cannot be overlooked. We demonstratethat this method of location based analysis is capable of delivering value added to established principlesof spatial planning in the age of trustworthy, large volume, heterogeneous data.

Solving Heterogeneous Big Data Mining Problems Using Multi-Objective Optimization

2019 ◽  
Vol 10 (4) ◽  
pp. 18-37
Author(s):  
Farid Bourennani
Keyword(s):  
Data Mining ◽  
Big Data ◽  
Research Area ◽  
Algebraic Model ◽  
Heterogeneous Data ◽  
Multi Objective Optimization ◽  
Data Types ◽  
Self Organizing Maps ◽  
Multi Objective ◽  
New Research

Nowadays, we have access to unprecedented quantities of data composed of heterogeneous data types (HDT). Heterogeneous data mining (HDM) is a new research area that focuses on the processing of HDT. Usually, input data is transformed into an algebraic model before data processing. However, how to combine the representations of HDT into a single model for a unified processing of big data is an open question. In this article, the authors attempt to find answers to this question by solving a data integration (DI) problem which involves the processing of seven HDT. They propose to solve the DI problem by combining multi-objective optimization and self-organizing maps to find optimal parameters settings for most accurate HDM results. The preliminary results are promising, and a post processing algorithm is proposed which makes the DI operations much simpler and more accurate.

De-Identification of Unstructured Textual Data using Artificial Immune System for Privacy Preserving

2016 ◽  
Vol 8 (4) ◽  
pp. 34-49 ◽  
Author(s):  
Amine Rahmani ◽  
Abdelmalek Amine ◽  
Reda Mohamed Hamou ◽  
Mohamed Amine Boudia ◽  
Hadj Ahmed Bouarara
Keyword(s):  
Big Data ◽  
High Efficiency ◽  
New Technologies ◽  
Artificial Immune Systems ◽  
Research Area ◽  
Heterogeneous Data ◽  
Small Scale ◽  
Artificial Immune ◽  
Critical Research ◽  
Textual Data

The development of new technologies has led the world into a tipping point. One of these technologies is the big data which made the revolution of computer sciences. Big data has come with new challenges. These challenges can be resumed in the aim of creating scalable and efficient services that can treat huge amounts of heterogeneous data in small scale of time while preserving users' privacy. Textual data occupy a wide space in internet. These data could contain information that can lead to identify users. For that, the development of such approaches that can detect and remove any identifiable information has become a critical research area known as de-identification. This paper tackle the problem of privacy in textual data. The authors' proposed approach consists of using artificial immune systems and MapReduce to detect and hide identifiable words with no matter on their variants using the personnel information of the user from his profile. After many experiments, the system shows a high efficiency in term of number of detected words, the way they are hided with, and time of execution.

Sign in / Sign up

Export Citation Format

Share Document