scholarly journals FONDUE: A Framework for Node Disambiguation and Deduplication Using Network Embeddings

2021 ◽  
Vol 11 (21) ◽  
pp. 9884
Author(s):  
Ahmad Mel ◽  
Bo Kang ◽  
Jefrey Lijffijt ◽  
Tijl De Bie
Keyword(s):  
Data Quality ◽  
State Of The Art ◽  
Computational Cost ◽  
Real Objects ◽  
Unweighted Network ◽  
Quality Issues ◽  
Main Components ◽  
Subsequent Splitting ◽  
Network Form ◽  
Embedding Methods

Data often have a relational nature that is most easily expressed in a network form, with its main components consisting of nodes that represent real objects and links that signify the relations between these objects. Modeling networks is useful for many purposes, but the efficacy of downstream tasks is often hampered by data quality issues related to their construction. In many constructed networks, ambiguity may arise when a node corresponds to multiple concepts. Similarly, a single entity can be mistakenly represented by several different nodes. In this paper, we formalize both the node disambiguation (NDA) and node deduplication (NDD) tasks to resolve these data quality issues. We then introduce FONDUE, a framework for utilizing network embedding methods for data-driven disambiguation and deduplication of nodes. Given an undirected and unweighted network, FONDUE-NDA identifies nodes that appear to correspond to multiple entities for subsequent splitting and suggests how to split them (node disambiguation), whereas FONDUE-NDD identifies nodes that appear to correspond to same entity for merging (node deduplication), using only the network topology. From controlled experiments on benchmark networks, we find that FONDUE-NDA is substantially and consistently more accurate with lower computational cost in identifying ambiguous nodes, and that FONDUE-NDD is a competitive alternative for node deduplication, when compared to state-of-the-art alternatives.

scholarly journals Research directions in data wrangling: Visualizations and transformations for usable and credible data

2011 ◽  
Vol 10 (4) ◽  
pp. 271-288 ◽  
Author(s):  
Sean Kandel ◽  
Jeffrey Heer ◽  
Catherine Plaisant ◽  
Jessie Kennedy ◽  
Frank van Ham ◽  
...  
Keyword(s):  
Data Quality ◽  
State Of The Art ◽  
Interactive Systems ◽  
Research Directions ◽  
Data Transformations ◽  
Challenges And Opportunities ◽  
Interactive Visualizations ◽  
Research Questions ◽  
Quality Issues ◽  
Integrate Data

In spite of advances in technologies for working with data, analysts still spend an inordinate amount of time diagnosing data quality issues and manipulating data into a usable form. This process of ‘data wrangling’ often constitutes the most tedious and time-consuming aspect of analysis. Though data cleaning and integration arelongstanding issues in the database community, relatively little research has explored how interactive visualization can advance the state of the art. In this article, we review the challenges and opportunities associated with addressing data quality issues. We argue that analysts might more effectively wrangle data through new interactive systems that integrate data verification, transformation, and visualization. We identify a number of outstanding research questions, including how appropriate visual encodings can facilitate apprehension of missing data, discrepant values, and uncertainty; how interactive visualizations might facilitate data transform specification; and how recorded provenance and social interaction might enable wider reuse, verification, and modification of data transformations.

scholarly journals Thinking about police data: Analysts’ perceptions of data quality in Canadian policing

2021 ◽  
pp. 0032258X2110214
Author(s):  
Christopher D O’Connor ◽  
John Ng ◽  
Dallas Hill ◽  
Tyler Frederick
Keyword(s):  
Big Data ◽  
Data Collection ◽  
Data Quality ◽  
Research Culture ◽  
Police Services ◽  
Police Data ◽  
Data Collection And Analysis ◽  
Quality Issues

Policing is increasingly being shaped by data collection and analysis. However, we still know little about the quality of the data police services acquire and utilize. Drawing on a survey of analysts from across Canada, this article examines several data collection, analysis, and quality issues. We argue that as we move towards an era of big data policing it is imperative that police services pay more attention to the quality of the data they collect. We conclude by discussing the implications of ignoring data quality issues and the need to develop a more robust research culture in policing.

scholarly journals Plant Leaf Disease Recognition Using Depth-Wise Separable Convolution-Based Models

Symmetry ◽  
2021 ◽  
Vol 13 (3) ◽  
pp. 511
Author(s):  
Syed Mohammad Minhaz Hossain ◽  
Kaushik Deb ◽  
Pranab Kumar Dhar ◽  
Takeshi Koshiba
Keyword(s):  
State Of The Art ◽  
Computational Cost ◽  
Region Of Interest ◽  
Number Of Clusters ◽  
Plant Leaf ◽  
Leaf Disease ◽  
Automatic Initialization ◽  
Adequate Accuracy ◽  
Model Size ◽  
High Computational Cost

Proper plant leaf disease (PLD) detection is challenging in complex backgrounds and under different capture conditions. For this reason, initially, modified adaptive centroid-based segmentation (ACS) is used to trace the proper region of interest (ROI). Automatic initialization of the number of clusters (K) using modified ACS before recognition increases tracing ROI’s scalability even for symmetrical features in various plants. Besides, convolutional neural network (CNN)-based PLD recognition models achieve adequate accuracy to some extent. However, memory requirements (large-scaled parameters) and the high computational cost of CNN-based PLD models are burning issues for the memory restricted mobile and IoT-based devices. Therefore, after tracing ROIs, three proposed depth-wise separable convolutional PLD (DSCPLD) models, such as segmented modified DSCPLD (S-modified MobileNet), segmented reduced DSCPLD (S-reduced MobileNet), and segmented extended DSCPLD (S-extended MobileNet), are utilized to represent the constructive trade-off among accuracy, model size, and computational latency. Moreover, we have compared our proposed DSCPLD recognition models with state-of-the-art models, such as MobileNet, VGG16, VGG19, and AlexNet. Among segmented-based DSCPLD models, S-modified MobileNet achieves the best accuracy of 99.55% and F1-sore of 97.07%. Besides, we have simulated our DSCPLD models using both full plant leaf images and segmented plant leaf images and conclude that, after using modified ACS, all models increase their accuracy and F1-score. Furthermore, a new plant leaf dataset containing 6580 images of eight plants was used to experiment with several depth-wise separable convolution models.

Between the Spreadsheets

2021 ◽  
Author(s):  
Susan Walsh
Keyword(s):  
Data Quality ◽  
Deep Understanding ◽  
Data Classification ◽  
Dirty Data ◽  
Quality Issues ◽  
Book Covers ◽  
Level Of Experience ◽  
The Impact

Dirty data is a problem that costs businesses thousands, if not millions, every year. In organisations large and small across the globe you will hear talk of data quality issues. What you will rarely hear about is the consequences or how to fix it.<br><br><i>Between the Spreadsheets: Classifying and Fixing Dirty Data</i> draws on classification expert Susan Walsh's decade of experience in data classification to present a fool-proof method for cleaning and classifying your data. The book covers everything from the very basics of data classification to normalisation, taxonomies and presents the author's proven <b>COAT</b> methodology, helping ensure an organisation's data is <b>Consistent</b>, <b>Organised</b>, <b>Accurate</b> and <b>Trustworthy</b>. A series of data horror stories outlines what can go wrong in managing data, and if it does, how it can be fixed. <br><br>After reading this book, regardless of your level of experience, not only will you be able to work with your data more efficiently, but you will also understand the impact the work you do with it has, and how it affects the rest of the organisation.<br><br>Written in an engaging and highly practical manner, <i>Between the Spreadsheets</i> gives readers of all levels a deep understanding of the dangers of dirty data and the confidence and skills to work more efficiently and effectively with it.

scholarly journals Data Quality: A Negotiator between Paper-based and Digital Records in the Pakistan’s TB Control Program

2018 ◽  
Author(s):  
Syed Mustafa Ali ◽  
Farah Naureen ◽  
Arif Noor ◽  
Maged Kamel N. Boulos ◽  
Javariya Aamir ◽  
...  
Keyword(s):  
Data Quality ◽  
Quality Assessment ◽  
Control Program ◽  
Digital Data ◽  
Patient Treatment ◽  
Assessment Framework ◽  
Healthcare Organizations ◽  
Data Quality Assessment ◽  
Quality Issues

Background Increasingly, healthcare organizations are using technology for the efficient management of data. The aim of this study was to compare the data quality of digital records with the quality of the corresponding paper-based records by using data quality assessment framework. Methodology We conducted a desk review of paper-based and digital records over the study duration from April 2016 to July 2016 at six enrolled TB clinics. We input all data fields of the patient treatment (TB01) card into a spreadsheet-based template to undertake a field-to-field comparison of the shared fields between TB01 and digital data. Findings A total of 117 TB01 cards were prepared at six enrolled sites, whereas just 50% of the records (n=59; 59 out of 117 TB01 cards) were digitized. There were 1,239 comparable data fields, out of which 65% (n=803) were correctly matched between paper based and digital records. However, 35% of the data fields (n=436) had anomalies, either in paper-based records or in digital records. 1.9 data quality issues were calculated per digital patient record, whereas it was 2.1 issues per record for paper-based record. Based on the analysis of valid data quality issues, it was found that there were more data quality issues in paper-based records (n=123) than in digital records (n=110). Conclusion There were fewer data quality issues in digital records as compared to the corresponding paper-based records. Greater use of mobile data capture and continued use of the data quality assessment framework can deliver more meaningful information for decision making.

scholarly journals OCR4all - An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings

2019 ◽  
Author(s):  
Christian Reul ◽  
Dennis Christ ◽  
Alexander Hartelt ◽  
Nico Balbach ◽  
Maximilian Wehner ◽  
...  
Keyword(s):  
Open Source ◽  
Character Recognition ◽  
Optical Character Recognition ◽  
State Of The Art ◽  
Continuous Model ◽  
Error Rates ◽  
Printed Books ◽  
Model Training ◽  
Easy Integration ◽  
Main Components

Optical Character Recognition (OCR) on historical printings is a challenging task mainly due to the complexity of the layout and the highly variant typography. Nevertheless, in the last few years great progress has been made in the area of historical OCR, resulting in several powerful open-source tools for preprocessing, layout recognition and segmentation, character recognition and post-processing. The drawback of these tools often is their limited applicability by non-technical users like humanist scholars and in particular the combined use of several tools in a workflow. In this paper we present an open-source OCR software called OCR4all, which combines state-of-the-art OCR components and continuous model training into a comprehensive workflow. A comfortable GUI allows error corrections not only in the final output, but already in early stages to minimize error propagations. Further on, extensive configuration capabilities are provided to set the degree of automation of the workflow and to make adaptations to the carefully selected default parameters for specific printings, if necessary. Experiments showed that users with minimal or no experience were able to capture the text of even the earliest printed books with manageable effort and great quality, achieving excellent character error rates (CERs) below 0.5%. The fully automated application on 19th century novels showed that OCR4all can considerably outperform the commercial state-of-the-art tool ABBYY Finereader on moderate layouts if suitably pretrained mixed OCR models are available. The architecture of OCR4all allows the easy integration (or substitution) of newly developed tools for its main components by standardized interfaces like PageXML, thus aiming at continual higher automation for historical printings.

scholarly journals Discovering XML Conditional Dependencies for Data Quality Issues

2020 ◽  
Vol 4 (1) ◽  
Author(s):  
Mohammed Ragheb Hakawati ◽  
Yasmin Yacob ◽  
Amiza Amir ◽  
Jabiry M. Mohammed ◽  
Khalid Jamal Jadaa
Keyword(s):  
Data Quality ◽  
Primary Standard ◽  
Markup Language ◽  
Document Type ◽  
Data Dependencies ◽  
Master Data ◽  
Xml Document ◽  
Extensible Markup ◽  
Quality Issues ◽  
Mining Algorithms

Extensible Markup Language (XML) is emerging as the primary standard for representing and exchanging data, with more than 60% of the total; XML considered the most dominant document type over the web; nevertheless, their quality is not as expected. XML integrity constraint especially XFD plays an important role in keeping the XML dataset as consistent as possible, but their ability to solve data quality issues is still intangible. The main reason is that old-fashioned data dependencies were basically introduced to maintain the consistency of the schema rather than that of the data. The purpose of this study is to introduce a method for discovering pattern tableaus for XML conditional dependencies to be used for enhancing XML document consistency as a part of data quality improvement phases. The notations of the conditional dependencies as new rules are designed mainly for improving data instance and extended traditional XML dependencies by enforcing pattern tableaus of semantically related constants. Subsequent to this, a set of minimal approximate conditional dependencies (XCFD, XCIND) is discovered and learned from the XML tree using a set of mining algorithms. The discovered patterns can be used as a Master data in order to detect inconsistencies that don’t respect the majority of the dataset.

scholarly journals On Dynamic Parallelization of Multilevel Monte Carlo Algorithm

2020 ◽  
Vol 20 (6) ◽  
pp. 116-125
Author(s):  
Nikolay Shegunov ◽  
Oleg Iliev
Keyword(s):  
Monte Carlo ◽  
Numerical Simulations ◽  
Random Fields ◽  
Stochastic Partial Differential Equations ◽  
Computational Cost ◽  
Monte Carlo Algorithm ◽  
Multilevel Monte Carlo ◽  
Deterministic Problem ◽  
Speed Up ◽  
Main Components

AbstractMultiLevel Monte Carlo (MLMC) attracts great interest for numerical simulations of Stochastic Partial Differential Equations (SPDEs), due to its superiority over the standard Monte Carlo (MC) approach. MLMC combines in a proper manner many cheap fast simulations with few slow and expensive ones, the variance is reduced, and a significant speed up is achieved. Simulations with MC/MLMC consist of three main components: generating random fields, solving deterministic problem and reduction of the variance. Each part is subject to a different degree of parallelism. Compared to the classical MC, MLMC introduces “levels” on which the sampling is done. These levels have different computational cost, thus, efficiently utilizing the parallel resources becomes a non-trivial problem. The main focus of this paper is the parallelization of the MLMC Algorithm.

Sign in / Sign up

Export Citation Format

Share Document