scholarly journals Secure Privacy Preserving Record Linkage of Large Databases by Modified Bloom Filter Encodings

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
Rainer Schnell ◽  
Christian Borgs
Keyword(s):  
Record Linkage ◽  
Large Scale ◽  
Bloom Filter ◽  
Privacy Preserving ◽  
Error Rates ◽  
Bloom Filters ◽  
Data Sets ◽  
Research Subjects ◽  
Practical Applications ◽  
Large Databases

ABSTRACTObjectiveIn most European settings, record linkage across different institutions has to be based on personal identifiers such as names, birthday or place of birth. To protect the privacy of research subjects, the identifiers have to be encrypted. In practice, these identifiers show error rates up to 20% per identifier, therefore linking on encrypted identifiers usually implies the loss of large subsets of the databases. In many applications, this loss of cases is related to variables of interest for the subject matter of the study. Therefore, this kind of record-linkage will generate biased estimates. These problems gave rise to techniques of Privacy Preserving Record Linkage (PPRL). Many different PPRL techniques have been suggested within the last 10 years, very few of them are suitable for practical applications with large database containing millions of records as they are typical for administrative or medical databases. One proven technique for PPRL for large scale applications is PPRL based on Bloom filters.MethodUsing appropriate parameter settings, Bloom filter approaches show linkage results comparable to linkage based on unencrypted identifiers. Furthermore, this approach has been used in real-world settings with data sets containing up to 100 Million records. By the application of suitable blocking strategies, linking can be done in reasonable time.ResultHowever, Bloom filters have been subject of cryptographic attacks. Previous research has shown that the straight application of Bloom filters has a nonzero re-identification risk. We will present new results on recently developed techniques to defy all known attacks on PPRL Bloom filters. These computationally simple algorithms modify the identifiers by different cryptographic diffusion techniques. The presentation will demonstrate these new algorithms and show their performance concerning precision, recall and re-identification risk on large databases.

scholarly journals Evaluating Binary Encoding Techniques in The Presence of Missing Values in Privacy-Preserving Record Linkage

2020 ◽  
Vol 5 (5) ◽  
Author(s):  
Thilina Ranbaduge ◽  
Peter Christen
Keyword(s):  
National Security ◽  
Record Linkage ◽  
Missing Values ◽  
Bloom Filter ◽  
Privacy Preserving ◽  
Bloom Filters ◽  
Practical Applications ◽  
Binary Encoding ◽  
Linkage Quality

IntroductionApplications in domains ranging from healthcare to national security increasingly require records about individuals in sensitive databases to be linked in privacy-preserving ways. Missing values make the linkage process challenging because they can affect the encoding of attribute values. No study has systematically investigated how missing values affect the outcomes of different encoding techniques used in privacy-preserving linkage applications. Objectives and ApproachBinary encodings, such as Bloom filters, are popular for linking sensitive databases. They are now employed in real-world linkage applications. However, existing encoding techniques assume the quasi-identifying attributes used for encoding to be complete. Missing values can lead to incomplete encodings which can result in decreased or increased similarities and therefore to false non-matches or false matches. In this study we empirically evaluate three binary encoding techniques using real voter databases, where pairs of records that correspond to the same voter (with name or address changes) resulted in files of 100,000 and 500,000 records containing from 0% to 50% missing values. ResultsWe encoded between two and four of the attributes first and last name, street, and city into three record-level binary encodings: Cryptographic long-term key (CLK) [Schnell et al. 2009], record-level Bloom filter (RBF) [Durham et al. 2014], and tabulation Min-hashing (TBH) [Smith 2017]. Experiments showed a 10% to 25% drop on average in both precision and recall for all encoding techniques when missing values are increasing. CLK resulted in the highest decrease in precision, while TBH resulted in the highest decrease in recall compared to the other encoding techniques. ConclusionBinary encodings such as Bloom filters are now used in practical applications for linking sensitive databases. Our evaluation shows that such encoding techniques can result in lower linkage quality if there are missing values in quasi-identifying attributes. This highlights the need for novel encoding techniques that can overcome the challenge of missing values.

scholarly journals A blinded evaluation of privacy preserving record linkage with Bloom filters

2022 ◽  
Vol 22 (1) ◽  
Author(s):  
Sean Randall ◽  
Helen Wichmann ◽  
Adrian Brown ◽  
James Boyd ◽  
Tom Eitelhuber ◽  
...  
Keyword(s):  
Record Linkage ◽  
Bloom Filter ◽  
Privacy Preserving ◽  
Bloom Filters ◽  
Filter Method ◽  
Morbidity Data ◽  
Linkage Quality ◽  
Hospital Morbidity ◽  
Western Australian ◽  
Direct Investigation

Abstract Background Privacy preserving record linkage (PPRL) methods using Bloom filters have shown promise for use in operational linkage settings. However real-world evaluations are required to confirm their suitability in practice. Methods An extract of records from the Western Australian (WA) Hospital Morbidity Data Collection 2011–2015 and WA Death Registrations 2011–2015 were encoded to Bloom filters, and then linked using privacy-preserving methods. Results were compared to a traditional, un-encoded linkage of the same datasets using the same blocking criteria to enable direct investigation of the comparison step. The encoded linkage was carried out in a blinded setting, where there was no access to un-encoded data or a ‘truth set’. Results The PPRL method using Bloom filters provided similar linkage quality to the traditional un-encoded linkage, with 99.3% of ‘groupings’ identical between privacy preserving and clear-text linkage. Conclusion The Bloom filter method appears suitable for use in situations where clear-text identifiers cannot be provided for linkage.

scholarly journals Cryptanalysis of Basic Bloom Filters Used for Privacy Preserving Record Linkage

2014 ◽  
Vol 6 (2) ◽  
Author(s):  
Frank Niedermeyer ◽  
Simone Steinmetzer ◽  
Martin Kroll ◽  
Rainer Schnell
Keyword(s):  
Statistical Analysis ◽  
Record Linkage ◽  
Bloom Filter ◽  
Privacy Preserving ◽  
Bloom Filters ◽  
Small Subset ◽  
Data Set ◽  
Successful Attack ◽  
Global Data

Bloom filter encoded identifiers are increasingly used for privacy preserving record linkage applications, because they allow for errors in encrypted identifiers. However, little research on the security of Bloom filters has been published so far. In this paper, we formalize a successful attack on Bloom filters composed of bigrams. It has previously been assumed in the literature that an attacker knows the global data set from which a sample is drawn. In contrast, we suppose that an attacker does not know this global data set. Instead, we assume the adversary knows a publicly available list of the most frequent attributes. The attack is based on subtle filtering and elementary statistical analysis of encrypted bigrams. The attack described in this paper can be used for the deciphering of a whole database instead of only a small subset of the most frequent names, as in previous research. We illustrate our proposed method with an attack on a database of encrypted surnames. Finally, we describe modifications of the Bloom filters for preventing similar attacks.

scholarly journals Large-scale sequence comparisons with sourmash

2019 ◽  
Author(s):  
N. Tessa Pierce ◽  
Luiz Irber ◽  
Taylor Reiter ◽  
Phillip Brooks ◽  
C. Titus Brown
Keyword(s):  
Software Package ◽  
Large Scale ◽  
Sequence Similarity ◽  
Protein Sequences ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Sequence Comparisons ◽  
Large Databases ◽  
Scale Sequence

The sourmash software package uses MinHash-based sketching to create “signatures”, compressed representations of DNA, RNA, and protein sequences, that can be stored, searched, explored, and taxonomically annotated. sourmash signatures can be used to estimate sequence similarity between very large data sets quickly and in low memory, and can be used to search large databases of genomes for matches to query genomes and metagenomes. sourmash is implemented in C++, Rust, and Python, and is freely available under the BSD license at http://github.com/dib-lab/sourmash.

scholarly journals Implementing privacy-preserving record linkage: welcome to the real world

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
James Boyd ◽  
Anna Ferrante ◽  
Adrian Brown ◽  
Sean Randall ◽  
James Semmens
Keyword(s):  
Data Integration ◽  
Real World ◽  
Error Detection ◽  
Record Linkage ◽  
Large Scale ◽  
Privacy Preserving ◽  
Estimation Methods ◽  
Experimental Conditions ◽  
Encrypted Data ◽  
Personally Identifying Information

ABSTRACT ObjectivesWhile record linkage has become a strategic research priority within Australia and internationally, legal and administrative issues prevent data linkage in some situations due to privacy concerns. Even current best practices in record linkage carry some privacy risk as they require the release of personally identifying information to trusted third parties. Application of record linkage systems that do not require the release of personal information can overcome legal and privacy issues surrounding data integration. Current conceptual and experimental privacy-preserving record linkage (PPRL) models show promise in addressing data integration challenges but do not yet address all of the requirements for real-world operations. This paper aims to identify and address some of the challenges of operationalising PPRL frameworks. ApproachTraditional linkage processes involve comparing personally identifying information (name, address, date of birth) on pairs of records to determine whether the records belong to the same person. Designing appropriate linkage strategies is an important part of the process. These are typically based on the analysis of data attributes (metadata) such as data completeness, consistency, constancy and field discriminating power. Under a PPRL model, however, these factors cannot be discerned from the encrypted data, so an alternative approach is required. This paper explores methods for data profiling, blocking, weight/threshold estimation and error detection within a PPRL framework. ResultsProbabilistic record linkage typically involves the estimation of weights and thresholds to optimise the linkage and ensure highly accurate results. The paper outlines the metadata requirements and automated methods necessary to collect data without compromising privacy. We present work undertaken to develop parameter estimation methods which can help optimise a linkage strategy without the release of personally identifiable information. These are required in all parts of the privacy preserving record linkage process (pre-processing, standardising activities, linkage, grouping and extracting). ConclusionsPPRL techniques that operate on encrypted data have the potential for large-scale record linkage, performing both accurately and efficiently under experimental conditions. Our research has advanced the current state of PPRL with a framework for secure record linkage that can be implemented to improve and expand linkage service delivery while protecting an individual’s privacy. However, more research is required to supplement this technique with additional elements to ensure the end-to-end method is practical and can be incorporated into real-world models.

scholarly journals Large-scale sequence comparisons with sourmash

F1000Research ◽  
2019 ◽  
Vol 8 ◽  
pp. 1006 ◽  
Author(s):  
N. Tessa Pierce ◽  
Luiz Irber ◽  
Taylor Reiter ◽  
Phillip Brooks ◽  
C. Titus Brown
Keyword(s):  
Software Package ◽  
Large Scale ◽  
Sequence Similarity ◽  
Protein Sequences ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Sequence Comparisons ◽  
Large Databases ◽  
Scale Sequence

The sourmash software package uses MinHash-based sketching to create “signatures”, compressed representations of DNA, RNA, and protein sequences, that can be stored, searched, explored, and taxonomically annotated. sourmash signatures can be used to estimate sequence similarity between very large data sets quickly and in low memory, and can be used to search large databases of genomes for matches to query genomes and metagenomes. sourmash is implemented in C++, Rust, and Python, and is freely available under the BSD license at http://github.com/dib-lab/sourmash.

Sign in / Sign up

Export Citation Format

Share Document