Hierarchical Linkage Clustering with Distributions of Distances for Large-Scale Record Linkage

2014 ◽  
pp. 283-298
Author(s):  
Samuel L. Ventura ◽  
Rebecca Nugent
Keyword(s):  
Record Linkage ◽  
Large Scale

The Use of Medical Record Linkage for Population and Genetic Studies

1969 ◽  
Vol 08 (01) ◽  
pp. 07-11 ◽  
Author(s):  
H. B. Newcombe
Keyword(s):  
Record Linkage ◽  
Large Scale ◽  
Medical Record Linkage ◽  
Canadian Province ◽  
Genetic Studies ◽  
Parental Characteristics ◽  
Family Histories ◽  
The Family ◽  
Large Populations ◽  
Machine Readable

Methods are described for deriving personal and family histories of birth, marriage, procreation, ill health and death, for large populations, from existing civil registrations of vital events and the routine records of ill health. Computers have been used to group together and »link« the separately derived records pertaining to successive events in the lives of the same individuals and families, rapidly and on a large scale. Most of the records employed are already available as machine readable punchcards and magnetic tapes, for statistical and administrative purposes, and only minor modifications have been made to the manner in which these are produced.As applied to the population of the Canadian province of British Columbia (currently about 2 million people) these methods have already yielded substantial information on the risks of disease: a) in the population, b) in relation to various parental characteristics, and c) as correlated with previous occurrences in the family histories.

Linked record medical information systems

1973 ◽  
Vol 184 (1077) ◽  
pp. 403-420 ◽  
Keyword(s):  
Information Systems ◽  
Record Linkage ◽  
Large Scale ◽  
Medical Information ◽  
Linkage Study ◽  
Computer Assisted ◽  
Medical Information Systems ◽  
Case Register ◽  
The Individual ◽  
Health Care Epidemiology

Linked record medical information systems generate cumulative person or family records from data on events or processes occurring to the individual at separate times and places. The special purposes of time-based and family statistics from these systems are exemplified by studies in health care, epidemiology and genetics. Methods of linking records on a large scale and some of the difficulties are described. The value and practicability of computer-assisted linked record systems are discussed in the light of experience with the Oxford Record Linkage Study and the northeast Scottish Psychiatric Case Register, both of which have been in existence for a decade.

scholarly journals Secure Privacy Preserving Record Linkage of Large Databases by Modified Bloom Filter Encodings

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
Rainer Schnell ◽  
Christian Borgs
Keyword(s):  
Record Linkage ◽  
Large Scale ◽  
Bloom Filter ◽  
Privacy Preserving ◽  
Error Rates ◽  
Bloom Filters ◽  
Data Sets ◽  
Research Subjects ◽  
Practical Applications ◽  
Large Databases

ABSTRACTObjectiveIn most European settings, record linkage across different institutions has to be based on personal identifiers such as names, birthday or place of birth. To protect the privacy of research subjects, the identifiers have to be encrypted. In practice, these identifiers show error rates up to 20% per identifier, therefore linking on encrypted identifiers usually implies the loss of large subsets of the databases. In many applications, this loss of cases is related to variables of interest for the subject matter of the study. Therefore, this kind of record-linkage will generate biased estimates. These problems gave rise to techniques of Privacy Preserving Record Linkage (PPRL). Many different PPRL techniques have been suggested within the last 10 years, very few of them are suitable for practical applications with large database containing millions of records as they are typical for administrative or medical databases. One proven technique for PPRL for large scale applications is PPRL based on Bloom filters.MethodUsing appropriate parameter settings, Bloom filter approaches show linkage results comparable to linkage based on unencrypted identifiers. Furthermore, this approach has been used in real-world settings with data sets containing up to 100 Million records. By the application of suitable blocking strategies, linking can be done in reasonable time.ResultHowever, Bloom filters have been subject of cryptographic attacks. Previous research has shown that the straight application of Bloom filters has a nonzero re-identification risk. We will present new results on recently developed techniques to defy all known attacks on PPRL Bloom filters. These computationally simple algorithms modify the identifiers by different cryptographic diffusion techniques. The presentation will demonstrate these new algorithms and show their performance concerning precision, recall and re-identification risk on large databases.

scholarly journals Implementing privacy-preserving record linkage: welcome to the real world

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
James Boyd ◽  
Anna Ferrante ◽  
Adrian Brown ◽  
Sean Randall ◽  
James Semmens
Keyword(s):  
Data Integration ◽  
Real World ◽  
Error Detection ◽  
Record Linkage ◽  
Large Scale ◽  
Privacy Preserving ◽  
Estimation Methods ◽  
Experimental Conditions ◽  
Encrypted Data ◽  
Personally Identifying Information

ABSTRACT ObjectivesWhile record linkage has become a strategic research priority within Australia and internationally, legal and administrative issues prevent data linkage in some situations due to privacy concerns. Even current best practices in record linkage carry some privacy risk as they require the release of personally identifying information to trusted third parties. Application of record linkage systems that do not require the release of personal information can overcome legal and privacy issues surrounding data integration. Current conceptual and experimental privacy-preserving record linkage (PPRL) models show promise in addressing data integration challenges but do not yet address all of the requirements for real-world operations. This paper aims to identify and address some of the challenges of operationalising PPRL frameworks. ApproachTraditional linkage processes involve comparing personally identifying information (name, address, date of birth) on pairs of records to determine whether the records belong to the same person. Designing appropriate linkage strategies is an important part of the process. These are typically based on the analysis of data attributes (metadata) such as data completeness, consistency, constancy and field discriminating power. Under a PPRL model, however, these factors cannot be discerned from the encrypted data, so an alternative approach is required. This paper explores methods for data profiling, blocking, weight/threshold estimation and error detection within a PPRL framework. ResultsProbabilistic record linkage typically involves the estimation of weights and thresholds to optimise the linkage and ensure highly accurate results. The paper outlines the metadata requirements and automated methods necessary to collect data without compromising privacy. We present work undertaken to develop parameter estimation methods which can help optimise a linkage strategy without the release of personally identifiable information. These are required in all parts of the privacy preserving record linkage process (pre-processing, standardising activities, linkage, grouping and extracting). ConclusionsPPRL techniques that operate on encrypted data have the potential for large-scale record linkage, performing both accurately and efficiently under experimental conditions. Our research has advanced the current state of PPRL with a framework for secure record linkage that can be implemented to improve and expand linkage service delivery while protecting an individual’s privacy. However, more research is required to supplement this technique with additional elements to ensure the end-to-end method is practical and can be incorporated into real-world models.

scholarly journals A Simple Sampling Method for Estimating the Accuracy of Large Scale Record Linkage Projects

2016 ◽  
Vol 55 (03) ◽  
pp. 276-283 ◽  
Author(s):  
Tenniel Guiver ◽  
Sean Randall ◽  
Anna Ferrante ◽  
James Semmens ◽  
Phil Anderson ◽  
...  
Keyword(s):  
Record Linkage ◽  
Large Scale ◽  
Sampling Method ◽  
Population Level ◽  
Kappa Statistics ◽  
False Negatives ◽  
True Match ◽  
Linkage Quality ◽  
Simple Sampling ◽  
Data Collections

SummaryBackground: Record linkage techniques allow different data collections to be brought together to provide a wider picture of the health status of individuals. Ensuring high linkage quality is important to guarantee the quality and integrity of research. Current methods for measuring linkage quality typically focus on precision (the proportion of incorrect links), given the difficulty of measuring the proportion of false negatives.Objectives: The aim of this work is to introduce and evaluate a sampling based method to estimate both precision and recall following record linkage.Methods: In the sampling based method, record-pairs from each threshold (including those below the identified cut-off for acceptance) are sampled and clerically reviewed. These results are then applied to the entire set of record-pairs, providing estimates of false positives and false negatives. This method was evaluated on a synthetically generated dataset, where the true match status (which records belonged to the same person) was known.Results: The sampled estimates of linkage quality were relatively close to actual linkage quality metrics calculated for the whole synthetic dataset. The precision and recall measures for seven reviewers were very consistent with little variation in the clerical assessment results (overall agreement using the Fleiss Kappa statistics was 0.601).Conclusions: This method presents as a possible means of accurately estimating matching quality and refining linkages in population level linkage studies. The sampling approach is especially important for large project linkages where the number of record pairs produced may be very large often running into millions.

INVITATION PAPER: THE INCIDENCE OF GENETIC DISEASE AND THE IMPACT ON MAN OF AN ALTERED MUTATION RATE

1977 ◽  
Vol 19 (3) ◽  
pp. 375-385 ◽  
Author(s):  
Benjamin K. Trimble ◽  
Martha E. Smith
Keyword(s):  
Mutation Rate ◽  
Record Linkage ◽  
Genetic Disease ◽  
Large Scale ◽  
Health Records ◽  
Individual Health ◽  
Naturally Occurring ◽  
The Social ◽  
Genetic Risks ◽  
The Impact

In order to adequately assess the genetic risks to man of an altered mutation rate, it is necessary to know the naturally occurring frequency of mutation-maintained genetic ill-health and the burden that such defects impose. The relevant data that are available are largely inadequate to "determine the incidence of genetic disease that is maintained by mutation, and measures of various aspects of the social and personal burdens due to hereditary ill-health are almost wholly lacking. It is suggested that the creation of individual and family histories, using large scale automatic record linkage and existing files of vital and ill-health records, may be a useful approach to these kinds of problems. Using such linked individual health histories, new data are presented that relate to measures of the burden due to childhood dominant and recessive diseases and congenital malformations.

scholarly journals Assessing the impact of different grouping methods: time to rethink and regroup?

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
Sean Randall ◽  
Anna Ferrante ◽  
Adrian Brown ◽  
James Boyd ◽  
James Semmens
Keyword(s):  
Record Linkage ◽  
Large Scale ◽  
Low Cost ◽  
Administrative Records ◽  
Grouping Strategy ◽  
Grouping Strategies ◽  
Linkage Quality ◽  
One To One ◽  
The Impact

ABSTRACT ObjectivesThe grouping of record-pairs to determine which administrative records belong to the same individual is an important process in record linkage. A variety of grouping methods are used but the relative benefits of each are unknown. We evaluate a number of grouping methods against the traditional merge based clustering approach using large scale administrative data. ApproachThe research aimed to both describe current grouping techniques used for record linkage, and to evaluate the most appropriate grouping method for specific circumstances. A range of grouping strategies were applied to three datasets with known truth sets. Conditions were simulated to appropriately investigate one-to-one, many-to-one and ongoing linkage scenarios. ResultsResults suggest alternate grouping methods will yield large benefits in linkage quality, especially when the quality of the underlying repository is high. Stepwise grouping methods were clearly superior for one-to-one linkage. There appeared little difference in linkage quality between many-to-one grouping approaches. The most appropriate techniques for ongoing linkage depended on the quality of the population spine and the underlying dataset. ConclusionsThese results demonstrate the large effect that the choice of grouping strategy can have on overall linkage quality. Ongoing linkages to high quality population spines provide large improvements in linkage quality compared to merge based linkages. Procuring or developing such a population spine will provide high linkage quality at far lower cost than current methods for improving linkage quality. By improving linkage quality at low cost, this resource can be further utilised by health researchers.

scholarly journals Integrating Collector and Author Roles Across Specimen and Publication Datasets

2019 ◽  
Vol 3 ◽  
Author(s):  
Nicky Nicolson ◽  
Alan Paton ◽  
Sarah Phillips ◽  
Allan Tucker
Keyword(s):  
Data Mining ◽  
Unsupervised Learning ◽  
Record Linkage ◽  
Large Scale ◽  
Single Source ◽  
Integration Process ◽  
Multiple Sources ◽  
Large Scale Dataset ◽  
Plant Names ◽  
International Plant

This work builds on the outputs of a collector data-mining exercise applied to GBIF mobilised herbarium specimen metadata, which uses unsupervised learning (clustering) to identify collectors from minimal metadata associated with field collected specimens (the DarwinCore terms recordedBy, eventDate and recordNumber). Here, we outline methods to integrate these data-mined collector entities (large scale dataset, aggregated from multiple sources, created programatically) with a dataset of author entities from the International Plant Names Index (smaller scale, single source dataset, created via editorial management). The integration process asserts a generic "scientist" entity with activities in different stages of the species description process: collecting and name publication. We present techniques to investigate specialisations including content - taxa of study - and activity stages: examining if individuals focus on collecting and/or name publication. Finally, we discuss generalisations of this initially herbarium-focussed data mining and record linkage process to enable applications in a wider context, particularly in zoological datasets.

scholarly journals Historical Census Record Linkage

2018 ◽  
Vol 44 (1) ◽  
pp. 19-37 ◽  
Author(s):  
Steven Ruggles ◽  
Catherine A. Fitch ◽  
Evan Roberts
Keyword(s):  
Record Linkage ◽  
Large Scale ◽  
Family Health ◽  
Later Life ◽  
Geographic Mobility ◽  
Natural Experiments ◽  
Data Infrastructure ◽  
Social Welfare Programs ◽  
The Impact ◽  
Census Record

For the past 80 years, social scientists have been linking historical censuses across time to study economic and geographic mobility. In recent decades, the quantity of historical census record linkage has exploded, owing largely to the advent of new machine-readable data created by genealogical organizations. Investigators are examining economic and geographic mobility across multiple generations and also engaging many new topics. Several analysts are exploring the effects of early-life socioeconomic conditions, environmental exposures, or natural disasters on family, health, and economic outcomes in later life. Other studies exploit natural experiments to gauge the impact of policy interventions such as social welfare programs and educational reforms. The new data sources have led to a proliferation of record linkage methodologies, and some widespread approaches inadvertently introduce errors that can lead to false inferences. A new generation of large-scale shared data infrastructure now in preparation will ameliorate weaknesses of current linkage methods.

Sign in / Sign up

Export Citation Format

Share Document