A Framework for Set Similarity Join on Multi-Attribute Data

Set similarity join, which finds all pairs of similar sets in a collection, plays an important role in data cleaning and integration. Many algorithms have been proposed to efficiently answer set similarity join on single-attribute data. However, real-world data often contain multiple attributes. In this paper, we propose a framework to enhance existing algorithms with additional filters for dealing with multi-attribute data. We then present a simple, yet effective filter based on lightweight indexes, for which exact and probabilistic implementation alternatives are evaluated. Finally, we devise a cost model to identify the best attribute ordering to reduce processing time. Our experimental results show that our approach is effective and significantly outperforms previous work.

Download Full-text

Efficient Set Similarity Join on Multi-Attribute Data Using Lightweight Filters

Journal of Information and Data Management ◽

10.5753/jidm.2021.1969 ◽

2021 ◽

Vol 12 (3) ◽

Author(s):

Leonardo Andrade Ribeiro ◽

Felipe Ferreira Borges ◽

Diego Oliveira

Keyword(s):

Data Structure ◽

Processing Time ◽

Cost Model ◽

Similarity Join ◽

Attribute Data ◽

Join Algorithms ◽

Filtering Technique ◽

Alternative Approaches ◽

Similarity Joins ◽

Single Set

We consider the problem of efficiently answering set similarity joins on multi-attribute data. Traditional set similarity join algorithms assume string data represented by a single set and, thus, miss the opportunity to exploit predicates over multiple attributes to reduce the number of similarity computations. In this article, we present a framework to enhance existing algorithms with additional filters for dealing with multi-attribute data. We then instantiate this framework with a lightweight filtering technique based on a simple, yet effective data structure, for which exact and probabilistic implementations are evaluated. In this context, we devise a cost model to identify the best attribute ordering to reduce processing time. Moreover, alternative approaches are also investigated and a new algorithm combining key ideas from previous work is introduced. Finally, we present a thorough experimental evaluation, which demonstrates that our main proposal is efficient and significantly outperforms competing algorithms.

Download Full-text

Enhancing of DBSCAN based on Sampling and Densitybased Separation

Iraqi Journal for Computers and Informatics ◽

10.25195/ijci.v42i1.82 ◽

2016 ◽

Vol 42 (1) ◽

pp. 38-47

Author(s):

Safaa Al-mamory ◽

Israa Kamil

Keyword(s):

Real World ◽

Clustering Algorithms ◽

Experimental Results ◽

New Technique ◽

Real World Data ◽

New Techniques ◽

World Data ◽

Density Based Clustering ◽

A New Technique

DBSCAN (Density-Based Clustering of Applications with Noise )is one of the attractive algorithms among densitybased clustering algorithms. It characterized by its ability to detect clusters of various sizes and shapes with the presence of noise, but its performance degrades when data have different densities .In this paper, we proposed a new technique to separate data based on its density with a new samplingtechnique , the purpose of these new techniques is for getting data with homogenous density .The experimental results onsynthetic data and real world data show that the new technique enhanced the clustering of DBSCAN to large extent.

Download Full-text

Inverted Bilingual Topic Models for Lexicon Extraction from Non-parallel Data

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2017/569 ◽

2017 ◽

Cited By ~ 1

Author(s):

Tengfei Ma ◽

Tetsuya Nasukawa

Keyword(s):

Real World ◽

Semantic Information ◽

Topic Models ◽

Experimental Results ◽

Real World Data ◽

Effective Measure ◽

World Data ◽

Parallel Data ◽

Multiple Translations

Topic models have been successfully applied in lexicon extraction. However, most previous methods are limited to document-aligned data. In this paper, we try to address two challenges of applying topic models to lexicon extraction in non-parallel data: 1) hard to model the word relationship and 2) noisy seed dictionary. To solve these two challenges, we propose two new bilingual topic models to better capture the semantic information of each word while discriminating the multiple translations in a noisy seed dictionary. We extend the scope of topic models by inverting the roles of "word" and "document". In addition, to solve the problem of noise in seed dictionary, we incorporate the probability of translation selection in our models. Moreover, we also propose an effective measure to evaluate the similarity of words in different languages and select the optimal translation pairs. Experimental results using real world data demonstrate the utility and efficacy of the proposed models.

Download Full-text

Venue-Influence Language Models for Expert Finding in Bibliometric Networks

International Journal on Semantic Web and Information Systems ◽

10.4018/ijswis.2018070109 ◽

2018 ◽

Vol 14 (3) ◽

pp. 184-201 ◽

Cited By ~ 3

Author(s):

Abdullah Al-Barakati ◽

Ali Daud

Keyword(s):

Real World ◽

Fundamental Problem ◽

Language Modeling ◽

Experimental Results ◽

Language Models ◽

Expert Finding ◽

Real World Data ◽

World Data ◽

Modeling Methods ◽

Traditional Language

This article investigates the fundamental problem of traditional language models used for expert finding in bibliometric networks. It introduces novel Venue-Influence Language Modeling methods based on entropy, which can accommodate citation links based weights in an indirect way without using links information. Intuitively, an author publishing in topic-specific venues, either journals or for conferences, will be an expert on a topic as compared to an author publishing in multi-topic venues. The proposed methods are evaluated on real world data, the Digital Bibliography and Library Project (DBLP) dataset to test the performance. Experimental results show that their proposed venue influence language models (ViLMs) based methods outperform the traditional (non-venue based) language models (LM).

Download Full-text