scholarly journals An approach to achieve high efficiency for large volume data processing using multiple clustering algorithms

2018 ◽  
Vol 7 (4.5) ◽  
pp. 689
Author(s):  
Sarada. B ◽  
Vinayaka Murthy. M ◽  
Udaya Rani. V
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Large Volume ◽  
High Efficiency ◽  
Clustering Algorithms ◽  
Computational Time ◽  
Sorting Algorithm ◽  
Data Sets ◽  
Volume Data ◽  
Data Points

Now a days data is increasing exponentially daily in terms of velocity, variety and volume which is also known as Big data. When the dataset has small number of dimensions, limited number of clusters and less number of data points the existing traditional clustering al- gorithms will give the expected results. As we know this is the Big Data age, with large volume of data sets through the traditional clus- tering algorithms we will not be able to get expected results. So there is a need to develop a new approach which gives better accuracy and computational time for large volume of data processing. The Proposed new System Architecture is a combination of canopy, Kmeans and RK sorting algorithm through Map Reduce Hadoop frame work platform. The analysis shows that the large volume of data processing will take less computational time and higher accuracy, and the RK sorting does not require swapping of elements and stack spaces. 

scholarly journals Analysis of large volume data processing using clustering algorithms

2018 ◽  
Vol 7 (4.5) ◽  
pp. 685
Author(s):  
Sarada. B ◽  
Vinayaka Murthy. M ◽  
Udaya Rani. V
Keyword(s):  
Data Processing ◽  
Large Volume ◽  
Clustering Algorithms ◽  
Volume Data ◽  
Spatial Complexity ◽  
New Approach ◽  
Low Dimensions ◽  
Internet Age ◽  
Data Points ◽  
Frame Work

The study of large dataset with velocity, variety and volume which is also known as Big data. When the dataset has limited number of clusters, low dimensions and small number of data points the existing traditional clustering algorithms can be used.. As we know this is the internet age, the data is growing very fast and existing clustering algorithms are not giving the acceptable results in terms of time complexity and spatial complexity. So there is a need to develop a new approach of applying clustering of large volume of data processing with low time and spatial complexity through MapReduce and Hadoop frame work applying to different clustering algorithms, k-means, Canopy clustering and proposed algorithm .The analysis shows that the large volume of data processing will take low time and spatial complexity when compared to small volume of data.   

scholarly journals Using Hadoop Technology to Overcome Big Data Problems by Choosing Proposed Cost-efficient Scheduler Algorithm for Heterogeneous Hadoop System (BD3)

2020 ◽  
pp. 58-84
Author(s):  
Abou_el_ela Abdou Hussein
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Data Storage ◽  
Database Management System ◽  
Data Sets ◽  
Complex Data ◽  
Daily Data ◽  
Complex Data Sets ◽  
Cost Efficient ◽  
Hadoop System

Day by day advanced web technologies have led to tremendous growth amount of daily data generated volumes. This mountain of huge and spread data sets leads to phenomenon that called big data which is a collection of massive, heterogeneous, unstructured, enormous and complex data sets. Big Data life cycle could be represented as, Collecting (capture), storing, distribute, manipulating, interpreting, analyzing, investigate and visualizing big data. Traditional techniques as Relational Database Management System (RDBMS) couldn’t handle big data because it has its own limitations, so Advancement in computing architecture is required to handle both the data storage requisites and the weighty processing needed to analyze huge volumes and variety of data economically. There are many technologies manipulating a big data, one of them is hadoop. Hadoop could be understand as an open source spread data processing that is one of the prominent and well known solutions to overcome handling big data problem. Apache Hadoop was based on Google File System and Map Reduce programming paradigm. Through this paper we dived to search for all big data characteristics starting from first three V's that have been extended during time through researches to be more than fifty six V's and making comparisons between researchers to reach to best representation and the precise clarification of all big data V’s characteristics. We highlight the challenges that face big data processing and how to overcome these challenges using Hadoop and its use in processing big data sets as a solution for resolving various problems in a distributed cloud based environment. This paper mainly focuses on different components of hadoop like Hive, Pig, and Hbase, etc. Also we institutes absolute description of Hadoop Pros and cons and improvements to face hadoop problems by choosing proposed Cost-efficient Scheduler Algorithm for heterogeneous Hadoop system.

Big Data Mining Based on Computational Intelligence and Fuzzy Clustering

2015 ◽  
pp. 130-148 ◽  
Author(s):  
Usman Akhtar ◽  
Mehdi Hassan
Keyword(s):  
Big Data ◽  
Computational Intelligence ◽  
Clustering Algorithms ◽  
Heterogeneous Data ◽  
Computational Time ◽  
Statistical Features ◽  
Feature Sets ◽  
Clustering Quality ◽  
The Given ◽  
Different Sources

The availability of a huge amount of heterogeneous data from different sources to the Internet has been termed as the problem of Big Data. Clustering is widely used as a knowledge discovery tool that separate the data into manageable parts. There is a need of clustering algorithms that scale on big databases. In this chapter we have explored various schemes that have been used to tackle the big databases. Statistical features have been extracted and most important and relevant features have been extracted from the given dataset. Reduce and irrelevant features have been eliminated and most important features have been selected by genetic algorithms (GA).Clustering with reduced feature sets requires lower computational time and resources. Experiments have been performed at standard datasets and results indicate that the proposed scheme based clustering offers high clustering accuracy. To check the clustering quality various quality measures have been computed and it has been observed that the proposed methodology results improved significantly. It has been observed that the proposed technique offers high quality clustering.

Big Data

2015 ◽  
pp. 326-344
Author(s):  
Piyush Kumar Shukla ◽  
Madhuvan Dixit
Keyword(s):  
Big Data ◽  
Large Volume ◽  
Knowledge Engineering ◽  
Complex Structure ◽  
Security And Privacy ◽  
Data Sets ◽  
User Interest ◽  
Image Information ◽  
Aggregation Of Information ◽  
Technical Area

In this chapter, Big Data provide large-volume, complex structure, heterogeneous and irregular growing data sets include multiple and autonomous different resources. In this chapter, With the growing improvement of networking sites, image information storing capacity become big issue too, Big Data concept are most growing expanding in all technical area and knowledge engineering domains, including physical, medical and paramedical sciences. Here a data-driven method consist demand-driven aggregation of information and knowledge mining and analysis, user interest prototyping, security and privacy aspects has been presented.

Big Data Mining Based on Computational Intelligence and Fuzzy Clustering

Web Services ◽  
2019 ◽  
pp. 413-430
Author(s):  
Usman Akhtar ◽  
Mehdi Hassan
Keyword(s):  
Big Data ◽  
Computational Intelligence ◽  
Clustering Algorithms ◽  
Heterogeneous Data ◽  
Computational Time ◽  
Statistical Features ◽  
Feature Sets ◽  
Clustering Quality ◽  
The Given ◽  
Different Sources

The availability of a huge amount of heterogeneous data from different sources to the Internet has been termed as the problem of Big Data. Clustering is widely used as a knowledge discovery tool that separate the data into manageable parts. There is a need of clustering algorithms that scale on big databases. In this chapter we have explored various schemes that have been used to tackle the big databases. Statistical features have been extracted and most important and relevant features have been extracted from the given dataset. Reduce and irrelevant features have been eliminated and most important features have been selected by genetic algorithms (GA). Clustering with reduced feature sets requires lower computational time and resources. Experiments have been performed at standard datasets and results indicate that the proposed scheme based clustering offers high clustering accuracy. To check the clustering quality various quality measures have been computed and it has been observed that the proposed methodology results improved significantly. It has been observed that the proposed technique offers high quality clustering.

Uncertainty-Based Clustering Algorithms for Large Data Sets

2018 ◽  
pp. 1-33 ◽  
Author(s):  
B. K. Tripathy ◽  
Hari Seetha ◽  
M. N. Murty
Keyword(s):  
Big Data ◽  
Data Clustering ◽  
Clustering Algorithms ◽  
Large Data ◽  
Large Data Sets ◽  
Mining Machine ◽  
Data Sets ◽  
Fuzzy C Means ◽  
Intuitionistic Fuzzy ◽  
New Algorithms

Data clustering plays a very important role in Data mining, machine learning and Image processing areas. As modern day databases have inherent uncertainties, many uncertainty-based data clustering algorithms have been developed in this direction. These algorithms are fuzzy c-means, rough c-means, intuitionistic fuzzy c-means and the means like rough fuzzy c-means, rough intuitionistic fuzzy c-means which base on hybrid models. Also, we find many variants of these algorithms which improve them in different directions like their Kernelised versions, possibilistic versions, and possibilistic Kernelised versions. However, all the above algorithms are not effective on big data for various reasons. So, researchers have been trying for the past few years to improve these algorithms in order they can be applied to cluster big data. The algorithms are relatively few in comparison to those for datasets of reasonable size. It is our aim in this chapter to present the uncertainty based clustering algorithms developed so far and proposes a few new algorithms which can be developed further.

The fast clustering algorithm for the big data based on K-means

2020 ◽  
Vol 18 (06) ◽  
pp. 2050053
Author(s):  
Ting Xie ◽  
Taiping Zhang
Keyword(s):  
Big Data ◽  
Clustering Algorithm ◽  
Clustering Algorithms ◽  
Feature Space ◽  
Data Sets ◽  
Benchmark Data ◽  
Clustering Model ◽  
Alternating Direction ◽  
Learning Technique ◽  
Noise Data

As a powerful unsupervised learning technique, clustering is the fundamental task of big data analysis. However, many traditional clustering algorithms for big data that is a collection of high dimension, sparse and noise data do not perform well both in terms of computational efficiency and clustering accuracy. To alleviate these problems, this paper presents Feature K-means clustering model on the feature space of big data and introduces its fast algorithm based on Alternating Direction Multiplier Method (ADMM). We show the equivalence of the Feature K-means model in the original space and the feature space and prove the convergence of its iterative algorithm. Computationally, we compare the Feature K-means with Spherical K-means and Kernel K-means on several benchmark data sets, including artificial data and four face databases. Experiments show that the proposed approach is comparable to the state-of-the-art algorithm in big data clustering.

SPSM: A NEW HYBRID DATA CLUSTERING ALGORITHM FOR NONLINEAR DATA ANALYSIS

2009 ◽  
Vol 23 (08) ◽  
pp. 1701-1737 ◽  
Author(s):  
UREERAT WATTANACHON ◽  
CHIDCHANOK LURSINSAP
Keyword(s):  
Clustering Algorithm ◽  
Color Image ◽  
Clustering Algorithms ◽  
Noisy Data ◽  
Second Phase ◽  
Data Sets ◽  
Data Set ◽  
Cluster Distance ◽  
Data Points ◽  
Hybrid Data

Existing clustering algorithms, such as single-link clustering, k-means, CURE, and CSM are designed to find clusters based on predefined parameters specified by users. These algorithms may be unsuccessful if the choice of parameters is inappropriate with respect to the data set being clustered. Most of these algorithms work very well for compact and hyper-spherical clusters. In this paper, a new hybrid clustering algorithm called Self-Partition and Self-Merging (SPSM) is proposed. The SPSM algorithm partitions the input data set into several subclusters in the first phase and, then, removes the noisy data in the second phase. In the third phase, the normal subclusters are continuously merged to form the larger clusters based on the inter-cluster distance and intra-cluster distance criteria. From the experimental results, the SPSM algorithm is very efficient to handle the noisy data set, and to cluster the data sets of arbitrary shapes of different density. Several examples for color image show the versatility of the proposed method and compare with results described in the literature for the same images. The computational complexity of the SPSM algorithm is O(N2), where N is the number of data points.

scholarly journals ClusterEnG: An interactive educational web resource for clustering big data

2017 ◽  
Author(s):  
Mohith Manjunath ◽  
Yi Zhang ◽  
Steve H. Yeo ◽  
Omar Sobh ◽  
Nathan Russell ◽  
...  
Keyword(s):  
Big Data ◽  
State Of The Art ◽  
Clustering Algorithms ◽  
Clustering Methods ◽  
Web Resource ◽  
Interactive Visualizations ◽  
Data Points ◽  
Similarities And Differences ◽  
Intuitive Manner ◽  
The Web

AbstractSummaryClustering is one of the most common techniques used in data analysis to discover hidden structures by grouping together data points that are similar in some measure into clusters. Although there are many programs available for performing clustering, a single web resource that provides both state-of-the-art clustering methods and interactive visualizations is lacking. ClusterEnG (acronym for Clustering Engine for Genomics) provides an interface for clustering big data and interactive visualizations including 3D views, cluster selection and zoom features. ClusterEnG also aims at educating the user about the similarities and differences between various clustering algorithms and provides clustering tutorials that demonstrate potential pitfalls of each algorithm. The web resource will be particularly useful to scientists who are not conversant with computing but want to understand the structure of their data in an intuitive manner.AvailabilityClusterEnG is part of a bigger project called KnowEnG (Knowledge Engine for Genomics) and is available at http://education.knoweng.org/[email protected]

Sign in / Sign up

Export Citation Format

Share Document