scholarly journals Attribute Reduction With Imputation Of Missing Data Using Fuzzy-Rougsh Set

2019 ◽  
Vol 8 (11) ◽  
pp. 202-207
Keyword(s):  
Data Mining ◽  
Missing Data ◽  
Attribute Reduction ◽  
Real Life ◽  
Data Imputation ◽  
Fuzzy Rough Set ◽  
Data Set ◽  
Missing Data Imputation ◽  
Real Life Data ◽  
Efficiency And Effectiveness

Attribute Reduction and missing data imputation have considerable influence in classification or other data mining task. New hybridization methodology like fuzzy rough set is more robust method to deal with imprecision and uncertainty for discrete as well as continuous data. Fuzzy rough attribute reduction with imputation (FRARI) algorithm has been proposed for attribute reduction with missing value imputation. So using FRARI algorithm complete reduce data set can be generated which has a great importance in different branches of artificial intelligence for data mining from databases. Efficiency and effectiveness of the proposed algorithm has been shown by experiment with real life data set.

Missing data imputation for multisite rainfall networks: a comparison between geostatistical interpolation and data-mining estimation on different terrain types

2020 ◽  
Author(s):  
Fabio Oriani ◽  
Simon Stisen ◽  
Mehmet C. Demirel ◽  
Gregoire Mariethoz
Keyword(s):  
Data Mining ◽  
Missing Data ◽  
Complex Structure ◽  
Physical Models ◽  
Data Driven ◽  
Training Dataset ◽  
Data Imputation ◽  
Missing Data Imputation ◽  
Mountainous Regions ◽  
Geostatistical Interpolation

<p>In the era of big data, missing data imputation remains a delicate topic for both the analysis of natural processes and to provide input data for physical models. We propose here a comparative study for missing data imputation on daily rainfall, a variable that can exhibit a complex structure composed of a dry/wet pattern and anisotropic sharp variations.</p><p>The seven algorithms considered can be grouped in two families: geostatistical interpolation techniques based on inverse-distance weighting and Kriging, widely used in gap-filling [1], and data-driven techniques based on the analysis of historical data patterns. This latter family of algorithms has been already applied to rainfall generation [2, 3], but it is not originally suitable to historical datasets presenting many data gaps. This happens because they usually operate in a rigid framework where, when a rainfall value is estimated for a station, the others are considered as predictor variables and require to be informed. To overcome this limitation, we propose here i) an adaptation of k-nearest neighbor (KNN) and ii) a new algorithm called Vector Sampling (VS), that combines concepts of multiple-point statistics and resampling. These data-driven algorithms can draw estimations from largely and variably incomplete data patterns, allowing the target dataset to be at the same time the training dataset.</p><p>Tested on different case studies from Denmark, Australia, and Switzerland, the algorithms show a different performance that seems to be related to the terrain type: on flat terrains with spatially uniform rain events, geostatistical interpolation tends to minimize the error, while, in mountainous regions with non-stationary rainfall statistics, data mining can recover better the complex rainfall patterns. The VS algorithm, being faster than KNN and requiring minimal parametrization, turns out to be a convenient option for routine application if a representative historical dataset is available. VS is open-source and freely available at .</p><p> </p><p>REFERENCES:</p><p></p><p><span>org/</span></p><p><span>org/</span></p>

scholarly journals Missing Data Imputation in High Dimensional Data Set using Local Similarity

2019 ◽  
Vol 8 (3) ◽  
pp. 8070-8074 ◽  
Keyword(s):  
Data Mining ◽  
Data Quality ◽  
Missing Values ◽  
Mean Squared Error ◽  
Clustering Algorithms ◽  
Local Similarity ◽  
Data Imputation ◽  
Data Set ◽  
Number Of Clusters ◽  
Missing Data Imputation

Data quality is an important aspect for any data mining and statistical tasks. Presence of missing values in the dataset affects the data quality. Missing values refers to the event did not happen or the value does not exist. Data mining algorithms are not robust towards incomplete data. Imputation of missing values is necessary to improve the data quality for performing data mining and statistical analysis. The existing methods such as Expectation Maximization Imputation (EMI), A Framework for Imputing Missing values Using co appearance, correlation and Similarity analysis (FIMUS) use the whole dataset to impute missing values. In such cases, due to the influence of irrelevant record the accuracy of imputation may be affected. This can be controlled by only considering locally similar records to impute missing values. Local similarity imputation can be done through clustering algorithms such as k-means algorithm. K-means clustering efficiency depends on the number of clusters is to be defined by users. To increase the clustering efficiency, first distinctive value is imputed in place of missing ones and this imputed dataset is given to stacked autoencoder for dimensionality reduction which also improves the efficiency of clustering. Initial number of clusters to k-means algorithm is determined using fast clustering. Due to initial imputation, some irrelevant records may be partitioned to a cluster. When these records are used for imputing missing values, accuracy of imputation decreases. In the proposed algorithm, local similarity imputation algorithm uses only top knearest neighbours within the cluster to impute missing values. The performance of the proposed algorithm is evaluated based on Root-Mean-Squared-Error (RMSE) and Index of Agreement (d2). University of California Irvine datasets has been used for analyzing the performance of the proposed algorithm.

Data Imputation Methods for Missing Values in the Context of Clustering

2019 ◽  
pp. 240-274
Author(s):  
Mehmet S. Aktaş ◽  
Sinan Kaplan ◽  
Hasan Abacı ◽  
Oya Kalipsiz ◽  
Utku Ketenci ◽  
...  
Keyword(s):  
Missing Data ◽  
Expectation Maximization ◽  
Missing Values ◽  
Nearest Neighbor ◽  
Real Life ◽  
Data Imputation ◽  
K Nearest Neighbor ◽  
Missing Data Imputation ◽  
Data Scarcity ◽  
Imputation Methods

Missing data is a common problem for data clustering quality. Most real-life datasets have missing data, which in turn has some effect on clustering tasks. This chapter investigates the appropriate data treatment methods for varying missing data scarcity distributions including gamma, Gaussian, and beta distributions. The analyzed data imputation methods include mean, hot-deck, regression, k-nearest neighbor, expectation maximization, and multiple imputation. To reveal the proper methods to deal with missing data, data mining tasks such as clustering is utilized for evaluation. With the experimental studies, this chapter identifies the correlation between missing data imputation methods and missing data distributions for clustering tasks. The results of the experiments indicated that expectation maximization and k-nearest neighbor methods provide best results for varying missing data scarcity distributions.

scholarly journals Likelihood Estimation with Incomplete Array Variate Observations

2014 ◽  
Author(s):  
Deniz Akdemir
Keyword(s):  
Missing Data ◽  
Probability Model ◽  
Real Life ◽  
Likelihood Estimation ◽  
Environment Interaction ◽  
Missing Data Imputation ◽  
Spectral Decompositions ◽  
Real Life Data ◽  
Normal Probability ◽  
Important Challenge

Missing data present an important challenge when dealing with high dimensional data arranged in the form of an array. In this paper, we propose methods for estimation of the parameters of array variate normal probability model from partially observed multi-way data. The methods developed here are useful for missing data imputation, estimation of mean and covariance parameters for multi-way data. A multi-way semi-parametric mixed effects model that allows separation of multi-way covariance effects is also defined and an efficient algorithm for estimation based on the spectral decompositions of the covariance parameters is recommended. We demonstrate our methods with simulations and with real life data involving the estimation of genotype and environment interaction effects on possibly correlated traits.

Sign in / Sign up

Export Citation Format

Share Document