scholarly journals A set theory based similarity measure for text clustering and classification

2020 ◽  
Vol 7 (1) ◽  
Author(s):  
Ali A. Amer ◽  
Hassan I. Abdalla
Keyword(s):  
Set Theory ◽  
Similarity Measure ◽  
Similarity Measures ◽  
Text Clustering ◽  
Plagiarism Detection ◽  
K Nearest Neighbor ◽  
Single Measure ◽  
Highly Effective ◽  
Clustering And Classification ◽  
Effectiveness And Efficiency

Abstract Similarity measures have long been utilized in information retrieval and machine learning domains for multi-purposes including text retrieval, text clustering, text summarization, plagiarism detection, and several other text-processing applications. However, the problem with these measures is that, until recently, there has never been one single measure recorded to be highly effective and efficient at the same time. Thus, the quest for an efficient and effective similarity measure is still an open-ended challenge. This study, in consequence, introduces a new highly-effective and time-efficient similarity measure for text clustering and classification. Furthermore, the study aims to provide a comprehensive scrutinization for seven of the most widely used similarity measures, mainly concerning their effectiveness and efficiency. Using the K-nearest neighbor algorithm (KNN) for classification, the K-means algorithm for clustering, and the bag of word (BoW) model for feature selection, all similarity measures are carefully examined in detail. The experimental evaluation has been made on two of the most popular datasets, namely, Reuters-21 and Web-KB. The obtained results confirm that the proposed set theory-based similarity measure (STB-SM), as a pre-eminent measure, outweighs all state-of-art measures significantly with regards to both effectiveness and efficiency.

scholarly journals Boolean logic algebra driven similarity measure for text based applications

2021 ◽  
Vol 7 ◽  
pp. e641
Author(s):  
Hassan I. Abdalla ◽  
Ali A. Amer
Keyword(s):  
Similarity Measure ◽  
Clustering Algorithm ◽  
Nearest Neighbor ◽  
Comprehensive Evaluation ◽  
State Of The Art ◽  
Similarity Measures ◽  
Boolean Logic ◽  
K Nearest Neighbor ◽  
Complex Design ◽  
Clustering And Classification

In Information Retrieval (IR), Data Mining (DM), and Machine Learning (ML), similarity measures have been widely used for text clustering and classification. The similarity measure is the cornerstone upon which the performance of most DM and ML algorithms is completely dependent. Thus, till now, the endeavor in literature for an effective and efficient similarity measure is still immature. Some recently-proposed similarity measures were effective, but have a complex design and suffer from inefficiencies. This work, therefore, develops an effective and efficient similarity measure of a simplistic design for text-based applications. The measure developed in this work is driven by Boolean logic algebra basics (BLAB-SM), which aims at effectively reaching the desired accuracy at the fastest run time as compared to the recently developed state-of-the-art measures. Using the term frequency–inverse document frequency (TF-IDF) schema, the K-nearest neighbor (KNN), and the K-means clustering algorithm, a comprehensive evaluation is presented. The evaluation has been experimentally performed for BLAB-SM against seven similarity measures on two most-popular datasets, Reuters-21 and Web-KB. The experimental results illustrate that BLAB-SM is not only more efficient but also significantly more effective than state-of-the-art similarity measures on both classification and clustering tasks.

An Improved Similarity Measure for Text Clustering and Classification

2015 ◽  
Vol 21 (11) ◽  
pp. 3583-3590 ◽  
Author(s):  
G Suresh Reddy ◽  
T. V Rajini Kanth ◽  
A Ananda Rao
Keyword(s):  
Similarity Measure ◽  
Text Clustering ◽  
Clustering And Classification

scholarly journals Similarity Measure and Entropy of Fuzzy Soft Sets

2014 ◽  
Vol 2014 ◽  
pp. 1-10 ◽  
Author(s):  
Zhicai Liu ◽  
Keyun Qin ◽  
Zheng Pei
Keyword(s):  
Set Theory ◽  
Similarity Measure ◽  
Similarity Measures ◽  
Soft Set ◽  
The Other ◽  
Mathematical Tool ◽  
Soft Sets ◽  
Uncertainty Measures ◽  
Fuzzy Soft Sets

Soft set theory, proposed by Molodtsov, has been regarded as an effective mathematical tool to deal with uncertainties. Recently, uncertainty measures of soft sets and fuzzy soft sets have gained attentions from researchers. This paper is devoted to the study of uncertainty measures of fuzzy soft sets. The axioms for similarity measure and entropy are proposed. A new category of similarity measures and entropies is presented based on fuzzy equivalence. Our approach is general in the sense that by using different fuzzy equivalences one gets different similarity measures and entropies. The relationships among these measures and the other proposals in the literatures are analyzed.

Evaluating Semantic Metrics on Tasks of Concept Similarity

2012 ◽  
pp. 324-340 ◽  
Author(s):  
Hansen A. Schwartz ◽  
Fernando Gomez
Keyword(s):  
Information Content ◽  
Similarity Measure ◽  
Similarity Measures ◽  
Major Conclusion ◽  
Single Measure ◽  
Word Similarity ◽  
The Difference ◽  
Word Senses ◽  
Concept Similarity ◽  
Semantic Metrics

In this study, first, concept similarity measures are evaluated over human judgments by using existing sets of word similarity pairs that we annotated with word senses. Next, an application-oriented study is presented to evaluate semantic metrics based on integration into an algorithm, first focused on the task of concept similarity then on the task of concept relatedness. The results found no single measure to be most significantly correlated with human-judgments, while an information content-based measure clearly lead to the best results in the application-oriented task of concept similarity. Reinforcing the difference between tasks of concept similarity and concept relatedness, the best measure for an application-oriented task of concept relatedness was a gloss-based relatedness measure rather than a similarity measure. A major conclusion of this work is that similarity measures may perform differently if embedded in specific applications than if they are compared with human judgments.

scholarly journals ONLINE FUZZY CLUSTERING OF INCOMPLETE DATA USING CREDIBILISTIC APPROACH AND SIMILARITY MEASURE OF SPECIAL TYPE

2021 ◽  
Vol 1 (1) ◽  
pp. 97-104
Author(s):  
Ye. V. Bodyanskiy ◽  
A. Yu. Shafronenko ◽  
I. N. Klymova
Keyword(s):  
Similarity Measure ◽  
Fuzzy Clustering ◽  
Incomplete Data ◽  
Similarity Measures ◽  
Real Data ◽  
Credibility Theory ◽  
Clustering And Classification ◽  
Clustering Data ◽  
Goal Functions ◽  
Distorted Data

Context. In most clustering (classification without a teacher) tasks associated with real data processing, the initial information is usually distorted by abnormal outliers (noise) and gaps. It is clear that “classical” methods of artificial intelligence (both batch and online) are ineffective in this situation.The goal of the paper is to propose the procedure of fuzzy clustering of incomplete data using credibilistic approach and similarity measure of special type. Objective. The goal of the work is credibilistic fuzzy clustering of distorted data, using of credibility theory. Method. The procedure of fuzzy clustering of incomplete data using credibilistic approach and similarity measure of special type based on the use of both robust goal functions of a special type and similarity measures, insensitive to outliers and designed to work both in batch and its recurrent online version designed to solve Data Stream Mining problems when data are fed to processing sequentially in real time. Results. The introduced methods are simple in numerical implementation and are free from the drawbacks inherent in traditional methods of probabilistic and possibilistic fuzzy clustering data distorted by abnormal outliers (noise) and gaps. Conclusions. The conducted experiments have confirmed the effectiveness of proposed methods of credibilistic fuzzy clustering of distorted data operability and allow recommending it for use in practice for solving the problems of automatic clusterization of distorted data. The proposed method is intended for use in hybrid systems of computational intelligence and, above all, in the problems of learning artificial neural networks, neuro-fuzzy systems, as well as in the problems of clustering and classification.

MATHURA (MBI) - A NOVEL IMPUTATION MEASURE FOR IMPUTATION OF MISSING VALUES IN MEDICAL DATASETS

2019 ◽  
Vol 13 ◽  
Author(s):  
B. Mathura Bai ◽  
N. Mangathayaru ◽  
B. Padmaja Rani ◽  
Shadi Aljawarneh
Keyword(s):  
Similarity Measure ◽  
Medical Records ◽  
Missing Values ◽  
Similarity Measures ◽  
Common Problems ◽  
Experiment Analysis

: Missing attribute values in medical datasets are one of the most common problems faced when mining medical datasets. Estimation of missing values is a major challenging task in pre-processing of datasets. Any wrong estimate of missing attribute values can lead to inefficient and improper classification thus resulting in lower classifier accuracies. Similarity measures play a key role during the imputation process. The use of an appropriate and better similarity measure can help to achieve better imputation and improved classification accuracies. This paper proposes a novel imputation measure for finding similarity between missing and non-missing instances in medical datasets. Experiments are carried by applying both the proposed imputation technique and popular benchmark existing imputation techniques. Classification is carried using KNN, J48, SMO and RBFN classifiers. Experiment analysis proved that after imputation of medical records using proposed imputation technique, the resulting classification accuracies reported by the classifiers KNN, J48 and SMO have improved when compared to other existing benchmark imputation techniques.

scholarly journals A New Approach to Measuring the Similarity of Indoor Semantic Trajectories

2021 ◽  
Vol 10 (2) ◽  
pp. 90
Author(s):  
Jin Zhu ◽  
Dayu Cheng ◽  
Weiwei Zhang ◽  
Ci Song ◽  
Jie Chen ◽  
...  
Keyword(s):  
Similarity Measure ◽  
Semantic Information ◽  
Edit Distance ◽  
Similarity Measures ◽  
Indoor Positioning ◽  
Synthetic Dataset ◽  
Shopping Mall ◽  
Indoor Space ◽  
Trajectory Similarity ◽  
Indoor Spaces

People spend more than 80% of their time in indoor spaces, such as shopping malls and office buildings. Indoor trajectories collected by indoor positioning devices, such as WiFi and Bluetooth devices, can reflect human movement behaviors in indoor spaces. Insightful indoor movement patterns can be discovered from indoor trajectories using various clustering methods. These methods are based on a measure that reflects the degree of similarity between indoor trajectories. Researchers have proposed many trajectory similarity measures. However, existing trajectory similarity measures ignore the indoor movement constraints imposed by the indoor space and the characteristics of indoor positioning sensors, which leads to an inaccurate measure of indoor trajectory similarity. Additionally, most of these works focus on the spatial and temporal dimensions of trajectories and pay less attention to indoor semantic information. Integrating indoor semantic information such as the indoor point of interest into the indoor trajectory similarity measurement is beneficial to discovering pedestrians having similar intentions. In this paper, we propose an accurate and reasonable indoor trajectory similarity measure called the indoor semantic trajectory similarity measure (ISTSM), which considers the features of indoor trajectories and indoor semantic information simultaneously. The ISTSM is modified from the edit distance that is a measure of the distance between string sequences. The key component of the ISTSM is an indoor navigation graph that is transformed from an indoor floor plan representing the indoor space for computing accurate indoor walking distances. The indoor walking distances and indoor semantic information are fused into the edit distance seamlessly. The ISTSM is evaluated using a synthetic dataset and real dataset for a shopping mall. The experiment with the synthetic dataset reveals that the ISTSM is more accurate and reasonable than three other popular trajectory similarities, namely the longest common subsequence (LCSS), edit distance on real sequence (EDR), and the multidimensional similarity measure (MSM). The case study of a shopping mall shows that the ISTSM effectively reveals customer movement patterns of indoor customers.

Knowledge Transfer for Entity Resolution with Siamese Neural Networks

2021 ◽  
Vol 13 (1) ◽  
pp. 1-25
Author(s):  
Michael Loster ◽  
Ioannis Koumarelas ◽  
Felix Naumann
Keyword(s):  
Knowledge Transfer ◽  
Similarity Measure ◽  
State Of The Art ◽  
Similarity Measures ◽  
Engineering Process ◽  
Domain Experts ◽  
Multiple Datasets ◽  
Multiple Data ◽  
Domain Expertise ◽  
F Measure

The integration of multiple data sources is a common problem in a large variety of applications. Traditionally, handcrafted similarity measures are used to discover, merge, and integrate multiple representations of the same entity—duplicates—into a large homogeneous collection of data. Often, these similarity measures do not cope well with the heterogeneity of the underlying dataset. In addition, domain experts are needed to manually design and configure such measures, which is both time-consuming and requires extensive domain expertise. We propose a deep Siamese neural network, capable of learning a similarity measure that is tailored to the characteristics of a particular dataset. With the properties of deep learning methods, we are able to eliminate the manual feature engineering process and thus considerably reduce the effort required for model construction. In addition, we show that it is possible to transfer knowledge acquired during the deduplication of one dataset to another, and thus significantly reduce the amount of data required to train a similarity measure. We evaluated our method on multiple datasets and compare our approach to state-of-the-art deduplication methods. Our approach outperforms competitors by up to +26 percent F-measure, depending on task and dataset. In addition, we show that knowledge transfer is not only feasible, but in our experiments led to an improvement in F-measure of up to +4.7 percent.

scholarly journals A Novel Cosine Similarity Like Data Clustering Method for Effective Data Classification in Data Mining

2020 ◽  
Vol 9 (8) ◽  
pp. 340-346
Keyword(s):  
Data Mining ◽  
Similarity Measure ◽  
Categorical Data ◽  
Data Clustering ◽  
Similarity Measures ◽  
Numerical Data ◽  
Data Classification ◽  
Fundamental Goal ◽  
Learning Technique ◽  
Categorical Data Clustering

In data mining ample techniques use distance based measures for data clustering. Improving clustering performance is the fundamental goal in cluster domain related tasks. Many techniques are available for clustering numerical data as well as categorical data. Clustering is an unsupervised learning technique and objects are grouped or clustered based on similarity among the objects. A new cluster similarity finding measure, which is cosine like cluster similarity measure (CLCSM), is proposed in this paper. The proposed cluster similarity measure is used for data classification. Extensive experiments are conducted by taking UCI machine learning datasets. The experimental results have shown that the proposed cosinelike cluster similarity measure is superior to many of the existing cluster similarity measures for data classification.

Sign in / Sign up

Export Citation Format

Share Document