scholarly journals Parallel Implementation of Improved K-Means Based on a Cloud Platform

2019 ◽  
Vol 48 (4) ◽  
pp. 673-681
Author(s):  
Shufen Zhang ◽  
Zhiyu Liu ◽  
Xuebin Chen ◽  
Changyin Luo
Keyword(s):  
Large Scale ◽  
Clustering Algorithm ◽  
Programming Model ◽  
Parallel Implementation ◽  
Clustering Algorithms ◽  
Data Set ◽  
Large Scale Data ◽  
Sample Density ◽  
Scale Data ◽  
Selection Of

In order to solve the problem of traditional K-Means clustering algorithm in dealing with large-scale data set, a Hadoop K-Means (referred to HKM) clustering algorithm is proposed. Firstly, according to the sample density, the algorithm eliminates the effects of noise points in the data set. Secondly, it optimizes the selection of the initial center point using the thought of the max-min distance. Finally, it uses a MapReduce programming model to realize the parallelization. Experimental results show that the proposed algorithm not only has high accuracy and stability in clustering results, but can also solve the problems of scalability encountered by traditional clustering algorithms in dealing with large scale data.

scholarly journals Clustering Large-Scale Data Based On Modified Affinity Propagation Algorithm

2016 ◽  
Vol 6 (1) ◽  
pp. 23-33 ◽  
Author(s):  
Ahmed M. Serdah ◽  
Wesam M. Ashour
Keyword(s):  
Data Clustering ◽  
Large Scale ◽  
Clustering Algorithm ◽  
Clustering Algorithms ◽  
Affinity Propagation ◽  
Clustering Method ◽  
Data Set ◽  
Local Cluster ◽  
Large Scale Data ◽  
Scale Data

Abstract Traditional clustering algorithms are no longer suitable for use in data mining applications that make use of large-scale data. There have been many large-scale data clustering algorithms proposed in recent years, but most of them do not achieve clustering with high quality. Despite that Affinity Propagation (AP) is effective and accurate in normal data clustering, but it is not effective for large-scale data. This paper proposes two methods for large-scale data clustering that depend on a modified version of AP algorithm. The proposed methods are set to ensure both low time complexity and good accuracy of the clustering method. Firstly, a data set is divided into several subsets using one of two methods random fragmentation or K-means. Secondly, subsets are clustered into K clusters using K-Affinity Propagation (KAP) algorithm to select local cluster exemplars in each subset. Thirdly, the inverse weighted clustering algorithm is performed on all local cluster exemplars to select well-suited global exemplars of the whole data set. Finally, all the data points are clustered by the similarity between all global exemplars and each data point. Results show that the proposed clustering method can significantly reduce the clustering time and produce better clustering result in a way that is more effective and accurate than AP, KAP, and HAP algorithms.

scholarly journals MapReduce-Based Crow Search-Adopted Partitional Clustering Algorithms for Handling Large-Scale Data

2021 ◽  
Vol 15 (4) ◽  
pp. 1-23
Author(s):  
Karthikeyani Visalakshi N. ◽  
Shanthi S. ◽  
Lakshmi K.
Keyword(s):  
Large Scale ◽  
Parallel Implementation ◽  
Search Algorithm ◽  
Clustering Algorithms ◽  
Computation Time ◽  
Global Optimum ◽  
Data Mining Technique ◽  
Local Optima ◽  
Large Scale Data ◽  
Scale Data

Cluster analysis is the prominent data mining technique in knowledge discovery and it discovers the hidden patterns from the data. The K-Means, K-Modes and K-Prototypes are partition based clustering algorithms and these algorithms select the initial centroids randomly. Because of its random selection of initial centroids, these algorithms provide the local optima in solutions. To solve these issues, the strategy of Crow Search algorithm is employed with these algorithms to obtain the global optimum solution. With the advances in information technology, the size of data increased in a drastic manner from terabytes to petabytes. To make proposed algorithms suitable to handle these voluminous data, the phenomena of parallel implementation of these clustering algorithms with Hadoop Mapreduce framework. The proposed algorithms are experimented with large scale data and the results are compared in terms of cluster evaluation measures and computation time with the number of nodes.

scholarly journals MapReduce Based Crow Search Adopted Partitional Clustering Algorithms For Handling Large Scale Data

2021 ◽  
Vol 15 (4) ◽  
pp. 0-0
Keyword(s):  
Large Scale ◽  
Parallel Implementation ◽  
Search Algorithm ◽  
Clustering Algorithms ◽  
Computation Time ◽  
Global Optimum ◽  
Data Mining Technique ◽  
Local Optima ◽  
Large Scale Data ◽  
Scale Data

Cluster analysis is the prominent data mining technique in knowledge discovery and it discovers the hidden patterns from the data. The K-Means, K-Modes and K-Prototypes are partition based clustering algorithms and these algorithms select the initial centroids randomly. Because of its random selection of initial centroids, these algorithms provide the local optima in solutions. To solve these issues, the strategy of Crow Search algorithm is employed with these algorithms to obtain the global optimum solution. With the advances in information technology, the size of data increased in a drastic manner from terabytes to petabytes. To make proposed algorithms suitable to handle these voluminous data, the phenomena of parallel implementation of these clustering algorithms with Hadoop Mapreduce framework. The proposed algorithms are experimented with large scale data and the results are compared in terms of cluster evaluation measures and computation time with the number of nodes.

Study of Map-Reduce over Hadoop Based Cloud Computing Environment

2014 ◽  
Vol 509 ◽  
pp. 175-181
Author(s):  
Wu Min Pan ◽  
Li Bai Ha
Keyword(s):  
Cloud Computing ◽  
Data Processing ◽  
Large Scale ◽  
Programming Model ◽  
Sql Server ◽  
Map Reduce ◽  
Data Set ◽  
Large Scale Data ◽  
Large Scale Data Processing ◽  
Scale Data

Popularity for the term Cloud-Computing has been increasing in recent years. In addition to the SQL technique, Map-Reduce, a programming model that realizes implementing large-scale data processing, has been a hot topic that is widely discussed through many studies. Many real-world tasks such as data processing for search engines can be parallel-implemented through a simple interface with two functions called Map and Reduce. We focus on comparing the performance of the Hadoop implementation of Map-Reduce with SQL Server through simulations. Hadoop can complete the same query faster than SQL Server. On the other hand, some concerned factors are also tested to see whether they would affect the performance for Hadoop or not. In fact more machines included for data processing can make Hadoop achieve a better performance, especially for a large-scale data set.

Landmark FN-DBSCAN: An Efficient Density-Based Clustering Algorithm with Fuzzy Neighborhood

2013 ◽  
Vol 17 (1) ◽  
pp. 60-73
Author(s):  
Hao Liu ◽  
◽  
Satoshi Oyama ◽  
Masahito Kurihara ◽  
Haruhiko Sato
Keyword(s):  
Time Complexity ◽  
Large Scale ◽  
Clustering Algorithm ◽  
Data Sets ◽  
Clustering Methods ◽  
Data Set ◽  
Large Scale Data ◽  
Density Based Clustering ◽  
Scale Data ◽  
Large Scale Data Sets

Clustering is an important tool for data analysis and many clustering techniques have been proposed over the past years. Among them are density-based clustering methods, which have several benefits such as the number of clusters is not required before carrying out clustering; the detected clusters can be represented in an arbitrary shape and outliers can be detected and removed. Recently, the density-based algorithms were extended with the fuzzy set theory, which has made these algorithm more robust. However, the density-based clustering algorithms usually require a time complexity ofO(n2) wherenis the number of data in the data set, implying that they are not suitable to work with large scale data sets. In this paper, a novel clustering algorithm called landmark fuzzy neighborhood DBSCAN (landmark FN-DBSCAN) is proposed. The concept, landmark, is used to represent a subset of the input data set which makes the algorithm efficient on large scale data sets. We give a theoretical analysis on time complexity and space complexity, which shows both of them are linear to the size of the data set. The experiments show that the landmark FN-DBSCAN is much faster than FN-DBSCAN and provides a very good quality of clustering.

ProGen:Provenance database generator for large-scale data set

2009 ◽  
Vol 28 (11) ◽  
pp. 2737-2740
Author(s):  
Xiao ZHANG ◽  
Shan WANG ◽  
Na LIAN
Keyword(s):  
Large Scale ◽  
Data Set ◽  
Large Scale Data ◽  
Scale Data
Sign in / Sign up

Export Citation Format

Share Document