Fast mining method of network heterogeneous fault tolerant data based on K-means clustering

2021 ◽  
pp. 1-10
Author(s):  
Haiyang Huang ◽  
Zhanlei Shang
Keyword(s):  
Data Model ◽  
Data Clustering ◽  
Euclidean Distance ◽  
Fault Tolerant ◽  
Mining Method ◽  
Redundant Data ◽  
Optimal Function ◽  
Clustering Center ◽  
Value Decomposition ◽  
Svd Method

In the traditional network heterogeneous fault-tolerant data mining process, there are some problems such as low accuracy and slow speed. This paper proposes a fast mining method based on K-means clustering for network heterogeneous fault-tolerant data. The confidence space of heterogeneous fault-tolerant data is determined, and the range of motion of fault-tolerant data is obtained; Singular value decomposition (SVD) method is used to construct the classified data model to obtain the characteristics of heterogeneous fault-tolerant data; The redundant data in fault-tolerant data is deleted by unsupervised feature selection algorithm, and the square sum and Euclidean distance of fault-tolerant data clustering center are determined by K-means algorithm. The discrete data clustering space is constructed, and the objective optimal function of network heterogeneous fault-tolerant data clustering is obtained, Realize fault-tolerant data fast mining. The results show that the mining accuracy of the proposed method can reach 97%.

scholarly journals Distributed Singular Value Decomposition Method for Fast Data Processing in Recommendation Systems

Energies ◽  
2021 ◽  
Vol 14 (8) ◽  
pp. 2284
Author(s):  
Krzysztof Przystupa ◽  
Mykola Beshley ◽  
Olena Hordiichuk-Bublivska ◽  
Marian Kyryk ◽  
Halyna Beshley ◽  
...  
Keyword(s):  
Distributed Systems ◽  
Singular Value Decomposition ◽  
Data Processing ◽  
Message Passing ◽  
Message Passing Interface ◽  
Recommendation Systems ◽  
Singular Value ◽  
Singular Value Decomposition Method ◽  
Value Decomposition ◽  
Svd Method

The problem of analyzing a big amount of user data to determine their preferences and, based on these data, to provide recommendations on new products is important. Depending on the correctness and timeliness of the recommendations, significant profits or losses can be obtained. The task of analyzing data on users of services of companies is carried out in special recommendation systems. However, with a large number of users, the data for processing become very big, which causes complexity in the work of recommendation systems. For efficient data analysis in commercial systems, the Singular Value Decomposition (SVD) method can perform intelligent analysis of information. With a large amount of processed information we proposed to use distributed systems. This approach allows reducing time of data processing and recommendations to users. For the experimental study, we implemented the distributed SVD method using Message Passing Interface, Hadoop and Spark technologies and obtained the results of reducing the time of data processing when using distributed systems compared to non-distributed ones.

scholarly journals Satellite image classification using proposed singular value decomposition method

2019 ◽  
Vol 13 (28) ◽  
pp. 52-67
Author(s):  
Noor Zubair Kouder
Keyword(s):  
Singular Value Decomposition ◽  
Classification Accuracy ◽  
Satellite Image ◽  
Field Work ◽  
Mean Value ◽  
Singular Value ◽  
Singular Value Decomposition Method ◽  
Erdas Imagine ◽  
Value Decomposition ◽  
Svd Method

In this work, satellite images for Razaza Lake and the surrounding areadistrict in Karbala province are classified for years 1990,1999 and2014 using two software programming (MATLAB 7.12 and ERDASimagine 2014). Proposed unsupervised and supervised method ofclassification using MATLAB software have been used; these aremean value and Singular Value Decomposition respectively. Whileunsupervised (K-Means) and supervised (Maximum likelihoodClassifier) method are utilized using ERDAS imagine, in order to getmost accurate results and then compare these results of each methodand calculate the changes that taken place in years 1999 and 2014;comparing with 1990. The results from classification indicated thatwater and hills are decreased, while vegetation, wet land and barrenland are increased for years 1999 and 2014; comparable with 1990.The classification accuracy was done by number of random pointschosen on the study area in the field work and geographical data thencompared with the classification results, the classification accuracy forthe proposed SVD method are 92.5%, 84.5% and 90% for years1990,1999,2014, respectivety, while the classification accuracies forunsupervised classification method based mean value are 92%, 87%and 91% for years 1990,1999,2014 respectivety.

A New Integrated Algorithm of Structure Determination and Parameter Estimation for Nonlinear Systems Identification

1999 ◽  
Author(s):  
Wang Xiao Wang ◽  
Jianyin Xie
Keyword(s):  
Parameter Estimation ◽  
Nonlinear Systems ◽  
Structure Determination ◽  
Model Identification ◽  
Identification Algorithm ◽  
Householder Transformation ◽  
Systems Identification ◽  
Value Decomposition ◽  
Svd Method ◽  
Nonlinear Systems Identification

Abstract A new integrated algorithm of structure determination and parameter estimation is proposed for nonlinear systems identification in this paper, which is based on the Householder Transformation (HT), Givens and Modified Gram-Schmidt (MGS) algorithms. While being used for the polynomial and rational NARMAX model identification, it can select the model terms while deleting the unimportant ones from the assumed full model, avoiding the storage difficulty as the CGS identification algorithm does which is proposed by Billings et. al., and is numerically more stable. Combining the H algorithm with the modified bidiagonalization least squares (MBLS) algorithm and the singular value decomposition (SVD) method respectively, two algorithms referred to as the MBLSHT and SVDHT ones are proposed for the polynomial and rational NARMAX model identification. They are all numerically more stable than the HT or Givens or MGS algorithm given in this paper, and the MBLSHT algorithm has the best performance. A higher precision for the parameter estimation can thus be obtained by them, as supported b simulation results.

scholarly journals SVD-based principal component analysis of geochemical data

2005 ◽  
Vol 3 (4) ◽  
pp. 731-741 ◽  
Author(s):  
Petr Praus
Keyword(s):  
Principal Component Analysis ◽  
Euclidean Distance ◽  
Principal Component ◽  
Component Analysis ◽  
Data Matrix ◽  
Geochemical Data ◽  
Average Group ◽  
Testing Data ◽  
Value Decomposition ◽  
Geochemical Variables

AbstractPrincipal Component Analysis (PCA) was used for the mapping of geochemical data. A testing data matrix was prepared from the chemical and physical analyses of the coals altered by thermal and oxidation effects. PCA based on Singular Value Decomposition (SVD) of the standardized (centered and scaled by the standard deviation) data matrix revealed three principal components explaining 85.2% of the variance. Combining the scatter and components weights plots with knowledge of the composition of tested samples, the coal samples were divided into seven groups depending on the degree of their oxidation and thermal alteration.The PCA findings were verified by other multivariate methods. The relationships among geochemical variables were successfully confirmed by Factor Analysis (FA). The data structure was also described by the Average Group dendrogram using Euclidean distance. The found sample clusters were not defined so clearly as in the case of PCA. It can be explained by the PCA filtration of the data noise.

scholarly journals Fault-Tolerant Control of a Distributed Database System

2008 ◽  
Vol 2008 ◽  
pp. 1-13 ◽  
Author(s):  
N. Eva Wu ◽  
Matthew C. Ruschmann ◽  
Mark H. Linderman
Keyword(s):  
Fault Tolerant ◽  
Control Policy ◽  
Distributed Database ◽  
Database System ◽  
Data Sets ◽  
Single Server ◽  
Markov Decision Problem ◽  
Redundant Data ◽  
Markov Decision ◽  
Distributed Database System

Optimal state information-based control policy for a distributed database system subject to server failures is considered. Fault-tolerance is made possible by the partitioned architecture of the system and data redundancy therein. Control actions include restoration of lost data sets in a single server using redundant data sets in the remaining servers, routing of queries to intact servers, or overhaul of the entire system for renewal. Control policies are determined by solving Markov decision problems with cost criteria that penalize system unavailability and slow query response. Steady-state system availability and expected query response time of the controlled database are evaluated with the Markov model of the database. Robustness is addressed by introducing additional states into the database model to account for control action delays and decision errors. A robust control policy is solved for the Markov decision problem described by the augmented state model.

scholarly journals Coordinate transformation parameters in Nepal by using neural network and SVD methods

2019 ◽  
Vol 9 (1) ◽  
pp. 22-28
Author(s):  
Kutubuddin Ansari ◽  
Prabin Gyawali ◽  
Prachand Man Pradhan ◽  
Kwan-Dong Park
Keyword(s):  
Neural Network ◽  
Coordinate Transformation ◽  
Data Sets ◽  
Extension Model ◽  
Global Positioning ◽  
Coordinate Measurements ◽  
Transformation Parameters ◽  
Artificial Neural Network Ann ◽  
Value Decomposition ◽  
Svd Method

Abstract The present study computes B-W extension model (extended Bursa-Wolf model) coordinate transformation parameters from World Geodetic System 1984 (WGS-84) to the Everest datum namely Everest (1830) and Everest (1956) using records of coordinate measurements from Global Positioning System (GPS) observable across Nepal region. Synthetic or modeled coordinates were determined by using the Artificial Neural Network (ANN) and Singular Value Decomposition (SVD) methods. We studied 9-transformation parameters with the help of the ANN technique and validated the outcomes with the SVD method. The comparative analysis of the ANN, as well as SVD methods, was done with the observed output following one way ANOVA test. The analysis showed that the null hypothesis for both datums were acceptable and suggesting all models statistically significantly equivalent to each other. The outcomes from this study would complement a relatively better understanding of the techniques for coordinate transformation and precise coordinate assignment while assimilating data sets from different resources.

scholarly journals Non-Negative K-SVD as an element of the forecasting electricity demand system

2019 ◽  
Vol 84 ◽  
pp. 01003
Author(s):  
Marcin Drechny
Keyword(s):  
Singular Value Decomposition ◽  
Sparse Coding ◽  
Demand Forecasting ◽  
Singular Value ◽  
Demand System ◽  
Power Demand ◽  
Power Load ◽  
Fast Power ◽  
Value Decomposition ◽  
Svd Method

The article describes the NN-K-SVD method based on the use of sparse coding and the singular value decomposition to specific values. An example of using the method is the compression of load profiles. The experiment of compression of 125022 power load profiles has been carried out with the use of registered profiles in households and small offices. Two matrices: patterns (atoms) and scaling factors are the result of the discussed algorithm. Features of the created matrices, which can be used in the creation of fast power demand forecasting systems, have been characterized.

scholarly journals Fuzzy Weighted Clustering Method for Numerical Attributes of Communication Big Data Based on Cloud Computing

Symmetry ◽  
2020 ◽  
Vol 12 (4) ◽  
pp. 530
Author(s):  
Haitao Ding ◽  
Chu Sun ◽  
Jianqiu Zeng
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Information Fusion ◽  
Data Clustering ◽  
Linear Regression Analysis ◽  
Numerical Attributes ◽  
Weighted Clustering ◽  
Feature Fitting ◽  
Feature Information ◽  
Clustering Center

It is necessary to optimize clustering processing of communication big data numerical attribute feature information in order to improve the ability of numerical attribute mining of communication big data, and thus a big data clustering algorithm based on cloud computing was proposed. The cloud extended distributed feature fitting method was used to process the numerical attribute linear programming of communication big data, and the mutual information feature quantity of communication big data numerical attribute was extracted. Combined with fuzzy C-means clustering and linear regression analysis, the statistical analysis of big data numerical attribute feature information was carried out, and the associated attribute sample set of communication big data numerical attribute cloud grid distribution was constructed. Cloud computing and adaptive quantitative recurrent classifiers were used for data classification, and block template matching and multi-sensor information fusion were combined to search the clustering center automatically to improve the convergence of clustering. The simulation results show that, after the application of this method, the information fusion performance of the clustering process was better, the automatic searching ability of the data clustering center was stronger, the frequency domain equalization control effect was good, the bit error rate was low, the energy consumption was small, and the ability of fuzzy weighted clustering retrieval of numerical attributes of communication big data was effectively improved.

Identification and Observability Measure of a Basis Set of Error Parameters in Robot Calibration

1989 ◽  
Vol 111 (4) ◽  
pp. 513-518 ◽  
Author(s):  
Chia-Hsiang Menq ◽  
Jin-Hwan Borm ◽  
Jim Z. Lai
Keyword(s):  
Basis Set ◽  
Robot Calibration ◽  
Position Errors ◽  
Estimation Of Error ◽  
Observability Index ◽  
Value Decomposition ◽  
Svd Method ◽  
Observability Measure ◽  
Robot Configurations ◽  
Error Parameter

This paper presents a method of identifying a basis set of error parameters in robot calibration using the Singular Value Decomposition (SVD) method. With the method, the error parameter space can be separated into two: observable subspace and unobservable one. As a result, for a defined position error model, one can determine the dimension of the observable subspace, which is vital to the estimation of error parameters. The second objective of this paper is to study, when unmodeled error exists, the implications of measurement configurations in robot calibration. For selecting measurement configurations in calibration, and index is defined to measure the observability of the error parameters with respect to a set of robot configurations. As the observability index increases, the attribution of the position errors to the parameters becomes dominant and the effects of the measurement and unmodeled errors become less significant; consequently better estimation of the parameter errors can be obtained.

scholarly journals The Shape From Motion Approach to Rapid and Precise Force/Torque Sensor Calibration

1997 ◽  
Vol 119 (2) ◽  
pp. 229-235 ◽  
Author(s):  
R. M. Voyles ◽  
J. D. Morrow ◽  
P. K. Khosla
Keyword(s):  
Least Squares ◽  
Explicit Knowledge ◽  
Calibration Procedure ◽  
Sensor Calibration ◽  
Torque Sensor ◽  
Data Set ◽  
Redundant Data ◽  
Sensor Signals ◽  
A New Technique ◽  
Value Decomposition

We present a new technique for multi-axis force/torque sensor calibration called shape from motion. The novel aspect of this technique is that it does not require explicit knowledge of the redundant applied load vectors, yet it retains the noise rejection of a highly redundant data set and the rigor of least squares. The result is a much faster, slightly more accurate calibration procedure. A constant-magnitude force (produced by a mass in a gravity field) is randomly moved through the sensing space while raw data is continuously gathered. Using only the raw sensor signals, the motion of the force vector (the “motion”) and the calibration matrix (the “shape”) are simultaneously extracted by singular value decomposition. We have applied this technique to several types of force/torque sensors and present experimental results for a 2-DOF fingertip and a 6-DOF wrist sensor with comparisons to the standard least squares approach.

Sign in / Sign up

Export Citation Format

Share Document