The Research on Large Scale Data Set Clustering Algorithm Based on Tag Set

In order to solve the problem of traditional K-Means clustering algorithm in dealing with large-scale data set, a Hadoop K-Means (referred to HKM) clustering algorithm is proposed. Firstly, according to the sample density, the algorithm eliminates the effects of noise points in the data set. Secondly, it optimizes the selection of the initial center point using the thought of the max-min distance. Finally, it uses a MapReduce programming model to realize the parallelization. Experimental results show that the proposed algorithm not only has high accuracy and stability in clustering results, but can also solve the problems of scalability encountered by traditional clustering algorithms in dealing with large scale data.

Download Full-text

Clustering Large-Scale Data Based On Modified Affinity Propagation Algorithm

Journal of Artificial Intelligence and Soft Computing Research ◽

10.1515/jaiscr-2016-0003 ◽

2016 ◽

Vol 6 (1) ◽

pp. 23-33 ◽

Cited By ~ 23

Author(s):

Ahmed M. Serdah ◽

Wesam M. Ashour

Keyword(s):

Data Clustering ◽

Large Scale ◽

Clustering Algorithm ◽

Clustering Algorithms ◽

Affinity Propagation ◽

Clustering Method ◽

Data Set ◽

Local Cluster ◽

Large Scale Data ◽

Scale Data

Abstract Traditional clustering algorithms are no longer suitable for use in data mining applications that make use of large-scale data. There have been many large-scale data clustering algorithms proposed in recent years, but most of them do not achieve clustering with high quality. Despite that Affinity Propagation (AP) is effective and accurate in normal data clustering, but it is not effective for large-scale data. This paper proposes two methods for large-scale data clustering that depend on a modified version of AP algorithm. The proposed methods are set to ensure both low time complexity and good accuracy of the clustering method. Firstly, a data set is divided into several subsets using one of two methods random fragmentation or K-means. Secondly, subsets are clustered into K clusters using K-Affinity Propagation (KAP) algorithm to select local cluster exemplars in each subset. Thirdly, the inverse weighted clustering algorithm is performed on all local cluster exemplars to select well-suited global exemplars of the whole data set. Finally, all the data points are clustered by the similarity between all global exemplars and each data point. Results show that the proposed clustering method can significantly reduce the clustering time and produce better clustering result in a way that is more effective and accurate than AP, KAP, and HAP algorithms.

Download Full-text

Affinity propagation clustering algorithm based on large-scale data-set

International Journal of Computers and Applications ◽

10.1080/1206212x.2018.1425184 ◽

2018 ◽

Vol 40 (3) ◽

pp. 1-6 ◽

Cited By ~ 4

Author(s):

Limin Wang ◽

Kaiyue Zheng ◽

Xing Tao ◽

Xuming Han

Keyword(s):

Large Scale ◽

Clustering Algorithm ◽

Affinity Propagation ◽

Data Set ◽

Large Scale Data ◽

Affinity Propagation Clustering ◽

Scale Data

Download Full-text

Landmark FN-DBSCAN: An Efficient Density-Based Clustering Algorithm with Fuzzy Neighborhood

Journal of Advanced Computational Intelligence and Intelligent Informatics ◽

10.20965/jaciii.2013.p0060 ◽

2013 ◽

Vol 17 (1) ◽

pp. 60-73

Author(s):

Hao Liu ◽

◽

Satoshi Oyama ◽

Masahito Kurihara ◽

Haruhiko Sato

Keyword(s):

Time Complexity ◽

Large Scale ◽

Clustering Algorithm ◽

Data Sets ◽

Clustering Methods ◽

Data Set ◽

Large Scale Data ◽

Density Based Clustering ◽

Scale Data ◽

Large Scale Data Sets

Clustering is an important tool for data analysis and many clustering techniques have been proposed over the past years. Among them are density-based clustering methods, which have several benefits such as the number of clusters is not required before carrying out clustering; the detected clusters can be represented in an arbitrary shape and outliers can be detected and removed. Recently, the density-based algorithms were extended with the fuzzy set theory, which has made these algorithm more robust. However, the density-based clustering algorithms usually require a time complexity ofO(n2) wherenis the number of data in the data set, implying that they are not suitable to work with large scale data sets. In this paper, a novel clustering algorithm called landmark fuzzy neighborhood DBSCAN (landmark FN-DBSCAN) is proposed. The concept, landmark, is used to represent a subset of the input data set which makes the algorithm efficient on large scale data sets. We give a theoretical analysis on time complexity and space complexity, which shows both of them are linear to the size of the data set. The experiments show that the landmark FN-DBSCAN is much faster than FN-DBSCAN and provides a very good quality of clustering.

Download Full-text