scholarly journals High-Value Token-Blocking: Efficient Blocking Method for Record Linkage

2021 ◽  
Vol 16 (2) ◽  
pp. 1-17
Author(s):  
Kevin O’hare ◽  
Anna Jurek-Loughrey ◽  
Cassio De Campos
Keyword(s):  
Data Integration ◽  
Real World ◽  
Record Linkage ◽  
Big Data Analytics ◽  
Heterogeneous Data ◽  
Heterogeneous Data Sources ◽  
Document Frequency ◽  
Scientific Papers ◽  
Computational Resources ◽  
Different Characteristics

Data integration is an important component of Big Data analytics. One of the key challenges in data integration is record linkage, that is, matching records that represent the same real-world entity. Because of computational costs, methods referred to as blocking are employed as a part of the record linkage pipeline in order to reduce the number of comparisons among records. In the past decade, a range of blocking techniques have been proposed. Real-world applications require approaches that can handle heterogeneous data sources and do not rely on labelled data. We propose high-value token-blocking (HVTB), a simple and efficient approach for blocking that is unsupervised and schema-agnostic, based on a crafted use of Term Frequency-Inverse Document Frequency. We compare HVTB with multiple methods and over a range of datasets, including a novel unstructured dataset composed of titles and abstracts of scientific papers. We thoroughly discuss results in terms of accuracy, use of computational resources, and different characteristics of datasets and records. The simplicity of HVTB yields fast computations and does not harm its accuracy when compared with existing approaches. It is shown to be significantly superior to other methods, suggesting that simpler methods for blocking should be considered before resorting to more sophisticated methods.

scholarly journals Integração, Relacionamento e Representação de Dados em Cidades Inteligentes: Uma Revisão de Literatura

2018 ◽  
Author(s):  
Larysse Silva ◽  
José Alex Lima ◽  
Nélio Cacho ◽  
Eiji Adachi ◽  
Frederico Lopes ◽  
...  
Keyword(s):  
Decision Making ◽  
Literature Review ◽  
Data Integration ◽  
Smart Cities ◽  
Heterogeneous Data ◽  
Data Sources ◽  
Application Development ◽  
Continuous Integration ◽  
Heterogeneous Data Sources ◽  
Computational Systems

A notable characteristic of smart cities is the increase in the amount of available data generated by several devices and computational systems, thus augmenting the challenges related to the development of software that involves the integration of larges volumes of data. In this context, this paper presents a literature review aimed to identify the main strategies used in the development of solutions for data integration, relationship, and representation in smart cities. This study systematically selected and analyzed eleven studies published from 2015 to 2017. The achieved results reveal gaps regarding solutions for the continuous integration of heterogeneous data sources towards supporting application development and decision-making.

Big Data Analytics for Tourism Destinations

2018 ◽  
pp. 349-363 ◽  
Author(s):  
Wolfram Höpken ◽  
Matthias Fuchs ◽  
Maria Lexhagen
Keyword(s):  
Data Analysis ◽  
Big Data Analytics ◽  
Heterogeneous Data ◽  
Machine Learning Techniques ◽  
Implicit Information ◽  
Heterogeneous Data Sources ◽  
Learning Techniques ◽  
Exploratory Data ◽  
Tourism Destinations ◽  
User Ratings

The objective of this chapter is to address the above deficiencies in tourism by presenting the concept of the tourism knowledge destination – a specific knowledge management architecture that supports value creation through enhanced supplier interaction and decision making. Information from heterogeneous data sources categorized into explicit feedback (e.g. tourist surveys, user ratings) and implicit information traces (navigation, transaction and tracking data) is extracted by applying semantic mapping, wrappers or text mining (Lau et al., 2005). Extracted data are stored in a central data warehouse enabling a destination-wide and all-stakeholder-encompassing data analysis approach. By using machine learning techniques interesting patterns are detected and knowledge is generated in the form of validated models (e.g. decision trees, neural networks, association rules, clustering models). These models, together with the underlying data (in the case of exploratory data analysis) are interactively visualized and made accessible to destination stakeholders.

Design and Implementation of Oilfield Heterogeneous Data Integration Model Based on Ontology

2014 ◽  
Vol 912-914 ◽  
pp. 1201-1204
Author(s):  
Gang Huang ◽  
Xiu Ying Wu ◽  
Man Yuan
Keyword(s):  
Data Integration ◽  
Heterogeneous Data ◽  
Data Sources ◽  
Semantic Heterogeneity ◽  
Integration Model ◽  
Integration Framework ◽  
Heterogeneous Data Integration ◽  
Semantic Level ◽  
Heterogeneous Data Sources ◽  
Semantic Difference

This paper provides an ontology-based distributed heterogeneous data integration framework (ODHDIF). The framework resolves the problem of semantic interoperability between heterogeneous data sources in semantic level. By metadatas specifying the distributed, heterogeneous data and by describing semantic information of data source , having "ontology" as a common semantic model, semantic match is established through ontology mapping between heterogeneous data sources and semantic difference institutions are shielded, so that semantic heterogeneity problem of the heterogeneous data sources can be effectively solved. It provides an effective technology measure for the interior information of enterprises to be shared in time accurately.

scholarly journals Implementing privacy-preserving record linkage: welcome to the real world

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
James Boyd ◽  
Anna Ferrante ◽  
Adrian Brown ◽  
Sean Randall ◽  
James Semmens
Keyword(s):  
Data Integration ◽  
Real World ◽  
Error Detection ◽  
Record Linkage ◽  
Large Scale ◽  
Privacy Preserving ◽  
Estimation Methods ◽  
Experimental Conditions ◽  
Encrypted Data ◽  
Personally Identifying Information

ABSTRACT ObjectivesWhile record linkage has become a strategic research priority within Australia and internationally, legal and administrative issues prevent data linkage in some situations due to privacy concerns. Even current best practices in record linkage carry some privacy risk as they require the release of personally identifying information to trusted third parties. Application of record linkage systems that do not require the release of personal information can overcome legal and privacy issues surrounding data integration. Current conceptual and experimental privacy-preserving record linkage (PPRL) models show promise in addressing data integration challenges but do not yet address all of the requirements for real-world operations. This paper aims to identify and address some of the challenges of operationalising PPRL frameworks. ApproachTraditional linkage processes involve comparing personally identifying information (name, address, date of birth) on pairs of records to determine whether the records belong to the same person. Designing appropriate linkage strategies is an important part of the process. These are typically based on the analysis of data attributes (metadata) such as data completeness, consistency, constancy and field discriminating power. Under a PPRL model, however, these factors cannot be discerned from the encrypted data, so an alternative approach is required. This paper explores methods for data profiling, blocking, weight/threshold estimation and error detection within a PPRL framework. ResultsProbabilistic record linkage typically involves the estimation of weights and thresholds to optimise the linkage and ensure highly accurate results. The paper outlines the metadata requirements and automated methods necessary to collect data without compromising privacy. We present work undertaken to develop parameter estimation methods which can help optimise a linkage strategy without the release of personally identifiable information. These are required in all parts of the privacy preserving record linkage process (pre-processing, standardising activities, linkage, grouping and extracting). ConclusionsPPRL techniques that operate on encrypted data have the potential for large-scale record linkage, performing both accurately and efficiently under experimental conditions. Our research has advanced the current state of PPRL with a framework for secure record linkage that can be implemented to improve and expand linkage service delivery while protecting an individual’s privacy. However, more research is required to supplement this technique with additional elements to ensure the end-to-end method is practical and can be incorporated into real-world models.

Data Integration Issues and Opportunities in Biological XML Data Management

2010 ◽  
pp. 263-286 ◽  
Author(s):  
Marco Mesiti ◽  
Ernesto Jiménez Ruiz ◽  
Ismael Sanz ◽  
Rafael Berlanga Llavori ◽  
Giorgio Valentini ◽  
...  
Keyword(s):  
Data Integration ◽  
Heterogeneous Data ◽  
Biological Data ◽  
Future Research ◽  
Data Types ◽  
Industrial Organizations ◽  
Heterogeneous Data Sources ◽  
Future Research Directions ◽  
Biological Data Integration ◽  
Xml Data Management

There is a proliferation of research and industrial organizations that produce sources of huge amounts of biological data issuing from experimentation with biological systems. In order to make these heterogeneous data sources easy to use, several efforts at data integration are currently being undertaken based mainly on XML. Starting from a discussion of the main biological data types and system interactions that need to be represented, the authors deal with the main approaches proposed for their modelling through XML. Then, they show the current efforts in biological data integration and how an increasing amount of Semantic information is required in terms of vocabulary control and ontologies. Finally, future research directions in biological data integration are discussed.

Research and Application on Oilfield Product Heterogeneous Data Integration Based on Ontology

2014 ◽  
Vol 530-531 ◽  
pp. 809-812
Author(s):  
Gang Huang ◽  
Xiu Ying Wu ◽  
Man Yuan ◽  
Rui Fang Li
Keyword(s):  
Data Integration ◽  
Heterogeneous Data ◽  
Data Sources ◽  
Distributed Data ◽  
Semantic Heterogeneity ◽  
Gas Industry ◽  
Heterogeneous Data Integration ◽  
Semantic Level ◽  
Heterogeneous Data Sources ◽  
Integrated Operations

The Oil & Gas industry is moving forward with Integrated Operations (IO). There are different ways to achieve data integration, and ontology-based approaches have drawn much attention. This paper introduces an ontology-based distributed data integration framework (ODDIF). The framework resolves the problem of semantic interoperability between heterogeneous data sources in semantic level. By metadatas specifying the distributed, heterogeneous data and by describing semantic information of data source , having "ontology" as a common semantic model, semantic match is established through ontology mapping between heterogeneous data sources and semantic difference institutions are shielded, so that semantic heterogeneity problem of the heterogeneous data sources can be effectively solved. The proposed method reduces developing difficulty, improves developing efficiency, and enhances the maintainability and expandability of the system.

Research on the Expert System for Sweet Corn Standard Production Based on Heterologous Data Integration Technology

2013 ◽  
Vol 655-657 ◽  
pp. 1730-1733
Author(s):  
Lin Peng ◽  
Qiang Zheng ◽  
Zhao Rong Liu
Keyword(s):  
Expert System ◽  
Data Integration ◽  
Data Management ◽  
Sweet Corn ◽  
Heterogeneous Data ◽  
Data Sources ◽  
Integration Technology ◽  
Shared Data ◽  
Heterogeneous Data Sources ◽  
Standard Production

To better share agricultural information in existed agricultural informatization condition, and to meet agro-departments new needs about local self-governed and global shared data management during standardized production of the sweet corn, this paper provides a method of integrated sharing of heterogeneous data sources to apply to standardized product of the sweet corn. This method solves the data integration and sharing problems during standardized production of the sweet corn. In this paper, the expert system for sweet corn standard production which is ability to combine heterogeneous data is constructed. This system is proved to be reliable, perform well and it is easy to operate.

scholarly journals An approach for semantic integration of heterogeneous data sources

2020 ◽  
Vol 6 ◽  
pp. e254
Author(s):  
Giuseppe Fusco ◽  
Lerina Aversano
Keyword(s):  
Data Integration ◽  
Heterogeneous Data ◽  
Semantic Integration ◽  
Data Sources ◽  
Complex Data ◽  
Semantic Heterogeneity ◽  
Heterogeneous Information ◽  
Heterogeneous Data Sources ◽  
Autonomous Data Sources ◽  
Unified View

Integrating data from multiple heterogeneous data sources entails dealing with data distributed among heterogeneous information sources, which can be structured, semi-structured or unstructured, and providing the user with a unified view of these data. Thus, in general, gathering information is challenging, and one of the main reasons is that data sources are designed to support specific applications. Very often their structure is unknown to the large part of users. Moreover, the stored data is often redundant, mixed with information only needed to support enterprise processes, and incomplete with respect to the business domain. Collecting, integrating, reconciling and efficiently extracting information from heterogeneous and autonomous data sources is regarded as a major challenge. In this paper, we present an approach for the semantic integration of heterogeneous data sources, DIF (Data Integration Framework), and a software prototype to support all aspects of a complex data integration process. The proposed approach is an ontology-based generalization of both Global-as-View and Local-as-View approaches. In particular, to overcome problems due to semantic heterogeneity and to support interoperability with external systems, ontologies are used as a conceptual schema to represent both data sources to be integrated and the global view.

Sign in / Sign up

Export Citation Format

Share Document