scholarly journals On the conditional distributions of low-dimensional projections from high-dimensional data

2013 ◽  
Vol 41 (2) ◽  
pp. 464-483 ◽  
Author(s):  
Hannes Leeb
Keyword(s):  
High Dimensional Data ◽  
High Dimensional ◽  
Conditional Distributions ◽  
Low Dimensional

scholarly journals A Novel Density-based Technique for Outlier Detection of High Dimensional Data Utilizing Full Feature Space

2021 ◽  
Vol 50 (1) ◽  
pp. 138-152
Author(s):  
Mujeeb Ur Rehman ◽  
Dost Muhammad Khan
Keyword(s):  
Data Mining ◽  
Outlier Detection ◽  
High Dimensional Data ◽  
Research Work ◽  
Feature Space ◽  
High Dimensional ◽  
Data Set ◽  
Data Points ◽  
Low Dimensional ◽  
Intrinsic Feature

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

scholarly journals Unsupervised Text Feature Learning via Deep Variational Auto-encoder

2020 ◽  
Vol 49 (3) ◽  
pp. 421-437
Author(s):  
Genggeng Liu ◽  
Lin Xie ◽  
Chi-Hua Chen
Keyword(s):  
Dimensionality Reduction ◽  
High Dimensional Data ◽  
Image Data ◽  
Original Data ◽  
Feature Representation ◽  
High Dimensional ◽  
Learning To Learn ◽  
Text Feature ◽  
Reduction Methods ◽  
Low Dimensional

Dimensionality reduction plays an important role in the data processing of machine learning and data mining, which makes the processing of high-dimensional data more efficient. Dimensionality reduction can extract the low-dimensional feature representation of high-dimensional data, and an effective dimensionality reduction method can not only extract most of the useful information of the original data, but also realize the function of removing useless noise. The dimensionality reduction methods can be applied to all types of data, especially image data. Although the supervised learning method has achieved good results in the application of dimensionality reduction, its performance depends on the number of labeled training samples. With the growing of information from internet, marking the data requires more resources and is more difficult. Therefore, using unsupervised learning to learn the feature of data has extremely important research value. In this paper, an unsupervised multilayered variational auto-encoder model is studied in the text data, so that the high-dimensional feature to the low-dimensional feature becomes efficient and the low-dimensional feature can retain mainly information as much as possible. Low-dimensional feature obtained by different dimensionality reduction methods are used to compare with the dimensionality reduction results of variational auto-encoder (VAE), and the method can be significantly improved over other comparison methods.

scholarly journals A System for Outlier Detection of High Dimensional Data

2012 ◽  
pp. 197-201
Author(s):  
Bharat Gupta ◽  
Durga Toshniwal
Keyword(s):  
Outlier Detection ◽  
High Dimensional Data ◽  
Research Problem ◽  
High Dimensional ◽  
Full Data ◽  
Data Set ◽  
Detection Techniques ◽  
New Concepts ◽  
Low Dimensional ◽  
Important Research Problem

In high dimensional data large no of outliers are embedded in low dimensional subspaces known as projected outliers, but most of existing outlier detection techniques are unable to find these projected outliers, because these methods perform detection of abnormal patterns in full data space. So, outlier detection in high dimensional data becomes an important research problem. In this paper we are proposing an approach for outlier detection of high dimensional data. Here we are modifying the existing SPOT approach by adding three new concepts namely Adaption of Sparse Sub-Space Template (SST), Different combination of PCS parameters and set of non outlying cells for testing data set.

scholarly journals Visual Exploration of Relationships and Structure in Low-Dimensional Embeddings

2021 ◽  
Author(s):  
Klaus Eckelt ◽  
Andreas Hinterreiter ◽  
Patrick Adelberger ◽  
Conny Walchshofer ◽  
Vaishali Dhanoa ◽  
...  
Keyword(s):  
High Dimensional Data ◽  
Visual Exploration ◽  
High Dimensional ◽  
Data Types ◽  
Structural Relationships ◽  
Or Groups ◽  
Analysis Workflow ◽  
Visual Approach ◽  
Real World Datasets ◽  
Low Dimensional

In this work, we propose an interactive visual approach for the exploration of structural relationships in embeddings of high-dimensional data. These structural relationships, such as item sequences, associations of items with groups, and hierarchies between groups of items, are defining properties of many real-world datasets. Nevertheless, most existing methods for the visual exploration of embeddings treat these structures as second-class citizens or do not take them into account at all. In our proposed analysis workflow, users explore enriched scatterplots of the embedding, in which relationships between items and/or groups are visually highlighted. The original high-dimensional data for single items, groups of items, or differences between connected items and groups is accessible through additional summary visualizations. We carefully tailored these summary and difference visualizations to the various data types and semantic contexts. During their exploratory analysis, users can externalize their insights by setting up additional groups and relationships between items and/or groups, thereby creating graphs that represent visual data stories. We demonstrate the utility and potential impact of our approach by means of two use cases and multiple examples from various domains.

scholarly journals A Preview on Subspace Clustering of High Dimensional Data

2013 ◽  
Vol 6 (3) ◽  
pp. 441-448 ◽  
Author(s):  
Sajid Nagi ◽  
Dhruba Kumar Bhattacharyya ◽  
Jugal K. Kalita
Keyword(s):  
Search Strategy ◽  
High Dimensional Data ◽  
Subspace Clustering ◽  
High Dimensional ◽  
Expression Data ◽  
Clustering Methods ◽  
Top Down ◽  
Data Points ◽  
Low Dimensional ◽  
Entire Dataset

When clustering high dimensional data, traditional clustering methods are found to be lacking since they consider all of the dimensions of the dataset in discovering clusters whereas only some of the dimensions are relevant. This may give rise to subspaces within the dataset where clusters may be found. Using feature selection, we can remove irrelevant and redundant dimensions by analyzing the entire dataset. The problem of automatically identifying clusters that exist in multiple and maybe overlapping subspaces of high dimensional data, allowing better clustering of the data points, is known as Subspace Clustering. There are two major approaches to subspace clustering based on search strategy. Top-down algorithms find an initial clustering in the full set of dimensions and evaluate the subspaces of each cluster, iteratively improving the results. Bottom-up approaches start from finding low dimensional dense regions, and then use them to form clusters. Based on a survey on subspace clustering, we identify the challenges and issues involved with clustering gene expression data.

scholarly journals M-Denclue for Effective Data Clustering in High Dimensional Non-Linear Data

2019 ◽  
Vol 9 (1) ◽  
pp. 2925-2927
Keyword(s):  
Clustering Algorithms ◽  
High Dimensional Data ◽  
Research Work ◽  
Curse Of Dimensionality ◽  
Distance Measures ◽  
High Dimensional ◽  
Clustering Methods ◽  
Non Linear ◽  
Low Dimensional ◽  
Automatic Grouping

Clustering is a data mining task devoted to the automatic grouping of data based on mutual similarity. Clustering in high-dimensional spaces is a recurrent problem in many domains. It affects time complexity, space complexity, scalability and accuracy of clustering methods. Highdimensional non-linear datausually live in different low dimensional subspaces hidden in the original space. As high‐dimensional objects appear almost alike, new approaches for clustering are required. This research has focused on developing Mathematical models, techniques and clustering algorithms specifically for high‐dimensional data. The innocent growth in the fields of communication and technology, there is tremendous growth in high dimensional data spaces. As the variant of dimensions on high dimensional non-linear data increases, many clustering techniques begin to suffer from the curse of dimensionality, de-grading the quality of the results. In high dimensional non-linear data, the data becomes very sparse and distance measures become increasingly meaningless. The principal challenge for clustering high dimensional data is to overcome the “curse of dimensionality”. This research work concentrates on devising an enhanced algorithm for clustering high dimensional non-linear data.

The Improvement of the CLIQUE Algorithm Based on High Dimensional Data Cleansing

2012 ◽  
Vol 452-453 ◽  
pp. 381-385
Author(s):  
Shao Peng Sun ◽  
Kai Hu Hou ◽  
Li Hua Chen
Keyword(s):  
Data Warehouse ◽  
High Dimensional Data ◽  
High Dimensional ◽  
Incremental Algorithms ◽  
Data Cleansing ◽  
Pruning Algorithm ◽  
Testing Data ◽  
Clique Algorithm ◽  
Abnormal Points ◽  
Low Dimensional

Many data cleansing algorithms are based on the low dimensional data currently, and can't meet the requirement of accuracy that data warehouse in the enterprise processes the high dimensional data. In this paper the idea of using the CLIQUE algorithm to process the high dimensional data was adopted. Aiming at the insufficient processing precision of this algorithm, the meshing and pruning algorithm were improved by using the dynamic incremental algorithms. The result of testing data shows that this algorithm can improve the accuracy of the clustering result and can effectively judge the similar clustering and abnormal points which support the high dimensional data cleansing.

Feature selection using autoencoders with Bayesian methods to high-dimensional data

2021 ◽  
pp. 1-10
Author(s):  
Lei Shu ◽  
Kun Huang ◽  
Wenhao Jiang ◽  
Wenming Wu ◽  
Hongling Liu
Keyword(s):  
Machine Learning ◽  
Feature Selection ◽  
Bayesian Methods ◽  
Large Scale ◽  
High Dimensional Data ◽  
Hybrid Approach ◽  
High Dimensional ◽  
Real World Data ◽  
Learning Tasks ◽  
Low Dimensional

It is easy to lead to poor generalization in machine learning tasks using real-world data directly, since such data is usually high-dimensional dimensionality and limited. Through learning the low dimensional representations of high-dimensional data, feature selection can retain useful features for machine learning tasks. Using these useful features effectively trains machine learning models. Hence, it is a challenge for feature selection from high-dimensional data. To address this issue, in this paper, a hybrid approach consisted of an autoencoder and Bayesian methods is proposed for a novel feature selection. Firstly, Bayesian methods are embedded in the proposed autoencoder as a special hidden layer. This of doing is to increase the precision during selecting non-redundant features. Then, the other hidden layers of the autoencoder are used for non-redundant feature selection. Finally, compared with the mainstream approaches for feature selection, the proposed method outperforms them. We find that the way consisted of autoencoders and probabilistic correction methods is more meaningful than that of stacking architectures or adding constraints to autoencoders as regards feature selection. We also demonstrate that stacked autoencoders are more suitable for large-scale feature selection, however, sparse autoencoders are beneficial for a smaller number of feature selection. We indicate that the value of the proposed method provides a theoretical reference to analyze the optimality of feature selection.

Sign in / Sign up

Export Citation Format

Share Document