Data Quality Issues in Data Integration Systems

2016 ◽  
pp. 279-307
Author(s):  
Carlo Batini ◽  
Monica Scannapieco
Keyword(s):  
Data Integration ◽  
Data Quality ◽  
Quality Issues

scholarly journals ETL Best Practices for Data Quality Checks in RIS Databases

Informatics ◽  
2019 ◽  
Vol 6 (1) ◽  
pp. 10 ◽  
Author(s):  
Otmane Azeroual ◽  
Gunter Saake ◽  
Mohammad Abuosba
Keyword(s):  
Data Integration ◽  
Data Quality ◽  
Information Quality ◽  
Database Systems ◽  
Data Sources ◽  
Research Information ◽  
External Data ◽  
Quality Checks ◽  
Quality Issues ◽  
It Departments

The topic of data integration from external data sources or independent IT-systems has received increasing attention recently in IT departments as well as at management level, in particular concerning data integration in federated database systems. An example of the latter are commercial research information systems (RIS), which regularly import, cleanse, transform and prepare the analysis research information of the institutions of a variety of databases. In addition, all these so-called steps must be provided in a secured quality. As several internal and external data sources are loaded for integration into the RIS, ensuring information quality is becoming increasingly challenging for the research institutions. Before the research information is transferred to a RIS, it must be checked and cleaned up. An important factor for successful or competent data integration is therefore always the data quality. The removal of data errors (such as duplicates and harmonization of the data structure, inconsistent data and outdated data, etc.) are essential tasks of data integration using extract, transform, and load (ETL) processes. Data is extracted from the source systems, transformed and loaded into the RIS. At this point conflicts between different data sources are controlled and solved, as well as data quality issues during data integration are eliminated. Against this background, our paper presents the process of data transformation in the context of RIS which gains an overview of the quality of research information in an institution’s internal and external data sources during its integration into RIS. In addition, the question of how to control and improve the quality issues during the integration process in RIS will be addressed.

Business Intelligence Architecture in Support of Data Quality

2014 ◽  
pp. 363-381
Author(s):  
Tom Breur
Keyword(s):  
Data Integration ◽  
Data Quality ◽  
Data Warehouse ◽  
Business Intelligence ◽  
Stakeholder Management ◽  
Source Data ◽  
Information Products ◽  
Quality Issues

Business Intelligence (BI) projects that involve substantial data integration have often proven failure-prone and difficult to plan. Data quality issues trigger rework, which makes it difficult to accurately schedule deliverables. Two things can bring improvement. Firstly, one should deliver information products in the smallest possible chunks, but without adding prohibitive overhead for breaking up the work in tiny increments. This will increase the frequency and improve timeliness of feedback on suitability of information products and hence make planning and progress more predictable. Secondly, BI teams need to provide better stewardship when they facilitate discussions between departments whose data cannot easily be integrated. Many so-called data quality errors do not stem from inaccurate source data, but rather from incorrect interpretation of data. This is mostly caused by different interpretation of essentially the same underlying source system facts across departments with misaligned performance objectives. Such problems require prudent stakeholder management and informed negotiations to resolve such differences. In this chapter, the authors suggest an innovation to data warehouse architecture to help accomplish these objectives.

scholarly journals Thinking about police data: Analysts’ perceptions of data quality in Canadian policing

2021 ◽  
pp. 0032258X2110214
Author(s):  
Christopher D O’Connor ◽  
John Ng ◽  
Dallas Hill ◽  
Tyler Frederick
Keyword(s):  
Big Data ◽  
Data Collection ◽  
Data Quality ◽  
Research Culture ◽  
Police Services ◽  
Police Data ◽  
Data Collection And Analysis ◽  
Quality Issues

Policing is increasingly being shaped by data collection and analysis. However, we still know little about the quality of the data police services acquire and utilize. Drawing on a survey of analysts from across Canada, this article examines several data collection, analysis, and quality issues. We argue that as we move towards an era of big data policing it is imperative that police services pay more attention to the quality of the data they collect. We conclude by discussing the implications of ignoring data quality issues and the need to develop a more robust research culture in policing.

A Power System Disturbance Classification Method Robust to PMU Data Quality Issues

2021 ◽  
pp. 1-1
Author(s):  
Zikang Li ◽  
Hao Liu ◽  
Junbo Zhao ◽  
Tianshu Bi ◽  
Qixun Yang
Keyword(s):  
Power System ◽  
Data Quality ◽  
Classification Method ◽  
Quality Issues

Between the Spreadsheets

2021 ◽  
Author(s):  
Susan Walsh
Keyword(s):  
Data Quality ◽  
Deep Understanding ◽  
Data Classification ◽  
Dirty Data ◽  
Quality Issues ◽  
Book Covers ◽  
Level Of Experience ◽  
The Impact

Dirty data is a problem that costs businesses thousands, if not millions, every year. In organisations large and small across the globe you will hear talk of data quality issues. What you will rarely hear about is the consequences or how to fix it.<br><br><i>Between the Spreadsheets: Classifying and Fixing Dirty Data</i> draws on classification expert Susan Walsh's decade of experience in data classification to present a fool-proof method for cleaning and classifying your data. The book covers everything from the very basics of data classification to normalisation, taxonomies and presents the author's proven <b>COAT</b> methodology, helping ensure an organisation's data is <b>Consistent</b>, <b>Organised</b>, <b>Accurate</b> and <b>Trustworthy</b>. A series of data horror stories outlines what can go wrong in managing data, and if it does, how it can be fixed. <br><br>After reading this book, regardless of your level of experience, not only will you be able to work with your data more efficiently, but you will also understand the impact the work you do with it has, and how it affects the rest of the organisation.<br><br>Written in an engaging and highly practical manner, <i>Between the Spreadsheets</i> gives readers of all levels a deep understanding of the dangers of dirty data and the confidence and skills to work more efficiently and effectively with it.

scholarly journals Data Quality: A Negotiator between Paper-based and Digital Records in the Pakistan’s TB Control Program

2018 ◽  
Author(s):  
Syed Mustafa Ali ◽  
Farah Naureen ◽  
Arif Noor ◽  
Maged Kamel N. Boulos ◽  
Javariya Aamir ◽  
...  
Keyword(s):  
Data Quality ◽  
Quality Assessment ◽  
Control Program ◽  
Digital Data ◽  
Patient Treatment ◽  
Assessment Framework ◽  
Healthcare Organizations ◽  
Data Quality Assessment ◽  
Quality Issues

Background Increasingly, healthcare organizations are using technology for the efficient management of data. The aim of this study was to compare the data quality of digital records with the quality of the corresponding paper-based records by using data quality assessment framework. Methodology We conducted a desk review of paper-based and digital records over the study duration from April 2016 to July 2016 at six enrolled TB clinics. We input all data fields of the patient treatment (TB01) card into a spreadsheet-based template to undertake a field-to-field comparison of the shared fields between TB01 and digital data. Findings A total of 117 TB01 cards were prepared at six enrolled sites, whereas just 50% of the records (n=59; 59 out of 117 TB01 cards) were digitized. There were 1,239 comparable data fields, out of which 65% (n=803) were correctly matched between paper based and digital records. However, 35% of the data fields (n=436) had anomalies, either in paper-based records or in digital records. 1.9 data quality issues were calculated per digital patient record, whereas it was 2.1 issues per record for paper-based record. Based on the analysis of valid data quality issues, it was found that there were more data quality issues in paper-based records (n=123) than in digital records (n=110). Conclusion There were fewer data quality issues in digital records as compared to the corresponding paper-based records. Greater use of mobile data capture and continued use of the data quality assessment framework can deliver more meaningful information for decision making.

Sign in / Sign up

Export Citation Format

Share Document