SeekAView: An intelligent dimensionality reduction strategy for navigating high-dimensional data spaces

Dimensionality reduction refers to a set of mathematical techniques used to reduce complexity of the original high-dimensional data, while preserving its selected properties. Improvements in simulation strategies and experimental data collection methods are resulting in a deluge of heterogeneous and high-dimensional data, which often makes dimensionality reduction the only viable way to gain qualitative and quantitative understanding of the data. However, existing dimensionality reduction software often does not scale to datasets arising in real-life applications, which may consist of thousands of points with millions of dimensions. In this paper, we propose a parallel framework for dimensionality reduction of large-scale data. We identify key components underlying the spectral dimensionality reduction techniques, and propose their efficient parallel implementation. We show that the resulting framework can be used to process datasets consisting of millions of points when executed on a 16,000-core cluster, which is beyond the reach of currently available methods. To further demonstrate applicability of our framework we perform dimensionality reduction of 75,000 images representing morphology evolution during manufacturing of organic solar cells in order to identify how processing parameters affect morphology evolution.

Download Full-text

Hierarchical Clustering of High-Dimensional Data Without Global Dimensionality Reduction

Lecture Notes in Computer Science - Foundations of Intelligent Systems ◽

10.1007/978-3-030-01851-1_23 ◽

2018 ◽

pp. 236-246

Author(s):

Ilari Kampman ◽

Tapio Elomaa

Keyword(s):

Dimensionality Reduction ◽

Hierarchical Clustering ◽

High Dimensional Data ◽

High Dimensional

Download Full-text

Unsupervised Text Feature Learning via Deep Variational Auto-encoder

Information Technology And Control ◽

10.5755/j01.itc.49.3.25918 ◽

2020 ◽

Vol 49 (3) ◽

pp. 421-437

Author(s):

Genggeng Liu ◽

Lin Xie ◽

Chi-Hua Chen

Keyword(s):

Dimensionality Reduction ◽

High Dimensional Data ◽

Image Data ◽

Original Data ◽

Feature Representation ◽

High Dimensional ◽

Learning To Learn ◽

Text Feature ◽

Reduction Methods ◽

Low Dimensional

Dimensionality reduction plays an important role in the data processing of machine learning and data mining, which makes the processing of high-dimensional data more efficient. Dimensionality reduction can extract the low-dimensional feature representation of high-dimensional data, and an effective dimensionality reduction method can not only extract most of the useful information of the original data, but also realize the function of removing useless noise. The dimensionality reduction methods can be applied to all types of data, especially image data. Although the supervised learning method has achieved good results in the application of dimensionality reduction, its performance depends on the number of labeled training samples. With the growing of information from internet, marking the data requires more resources and is more difficult. Therefore, using unsupervised learning to learn the feature of data has extremely important research value. In this paper, an unsupervised multilayered variational auto-encoder model is studied in the text data, so that the high-dimensional feature to the low-dimensional feature becomes efficient and the low-dimensional feature can retain mainly information as much as possible. Low-dimensional feature obtained by different dimensionality reduction methods are used to compare with the dimensionality reduction results of variational auto-encoder (VAE), and the method can be significantly improved over other comparison methods.

Download Full-text

A sparse grid based method for generative dimensionality reduction of high-dimensional data

Journal of Computational Physics ◽

10.1016/j.jcp.2015.12.033 ◽

2016 ◽

Vol 309 ◽

pp. 1-17 ◽

Cited By ~ 8

Author(s):

Bastian Bohn ◽

Jochen Garcke ◽

Michael Griebel

Keyword(s):

Dimensionality Reduction ◽

High Dimensional Data ◽

Sparse Grid ◽

High Dimensional ◽

Grid Based

Download Full-text

Glyphboard: Visual Exploration of High-Dimensional Data Combining Glyphs with Dimensionality Reduction

IEEE Transactions on Visualization and Computer Graphics ◽

10.1109/tvcg.2020.2969060 ◽

2020 ◽

Vol 26 (4) ◽

pp. 1661-1671 ◽

Cited By ~ 1

Author(s):

Dietrich Kammer ◽

Mandy Keck ◽

Thomas Grunder ◽

Alexander Maasch ◽

Thomas Thom ◽

...

Keyword(s):

Dimensionality Reduction ◽

High Dimensional Data ◽

Visual Exploration ◽

High Dimensional

Download Full-text

An Efficient Dimensionality Reduction Approach for Small-sample Size and High-dimensional Data Modeling

Journal of Computers ◽

10.4304/jcp.9.3.576-580 ◽

2014 ◽

Vol 9 (3) ◽

Cited By ~ 5

Author(s):

Xintao Qiu ◽

Dongmei Fu ◽

Zhenduo Fu

Keyword(s):

Dimensionality Reduction ◽

Sample Size ◽

Small Sample Size ◽

High Dimensional Data ◽

Data Modeling ◽

Small Sample ◽

High Dimensional ◽

Reduction Approach

Download Full-text

Dimensionality Reduction by Weighted Connections between Neighborhoods

Abstract and Applied Analysis ◽

10.1155/2014/928136 ◽

2014 ◽

Vol 2014 ◽

pp. 1-5 ◽

Cited By ~ 1

Author(s):

Fuding Xie ◽

Yutao Fan ◽

Ming Zhou

Keyword(s):

Dimensionality Reduction ◽

Dimensional Space ◽

High Dimensional Data ◽

Reduction Technique ◽

Experimental Results ◽

High Dimensional ◽

Reduced Dimensionality ◽

Dimensionality Reduction Technique ◽

Low Dimensionality ◽

Local Topology

Dimensionality reduction is the transformation of high-dimensional data into a meaningful representation of reduced dimensionality. This paper introduces a dimensionality reduction technique by weighted connections between neighborhoods to improveK-Isomap method, attempting to preserve perfectly the relationships between neighborhoods in the process of dimensionality reduction. The validity of the proposal is tested by three typical examples which are widely employed in the algorithms based on manifold. The experimental results show that the local topology nature of dataset is preserved well while transforming dataset in high-dimensional space into a new dataset in low-dimensionality by the proposed method.

Download Full-text

H-tSNE: Hierarchical Nonlinear Dimensionality Reduction

10.1101/2020.10.05.324798 ◽

2020 ◽

Author(s):

Kevin C. VanHorn ◽

Murat Can Çobanoğlu

Keyword(s):

Dimensionality Reduction ◽

Social Networking ◽

A Priori ◽

High Dimensional Data ◽

High Order ◽

High Dimensional ◽

Nonlinear Dimensionality Reduction ◽

Embedding Technique ◽

Class Labels ◽

Parent Child Relationships

AbstractDimensionality reduction (DR) is often integral when analyzing high-dimensional data across scientific, economic, and social networking applications. For data with a high order of complexity, nonlinear approaches are often needed to identify and represent the most important components. We propose a novel DR approach that can incorporate a known underlying hierarchy. Specifically, we extend the widely used t-Distributed Stochastic Neighbor Embedding technique (t-SNE) to include hierarchical information and demonstrate its use with known or unknown class labels. We term this approach “H-tSNE.” Such a strategy can aid in discovering and understanding underlying patterns of a dataset that is heavily influenced by parent-child relationships. Without integrating information that is known a priori, we suggest that DR cannot function as effectively. In this regard, we argue for a DR approach that enables the user to incorporate known, relevant relationships even if their representation is weakly expressed in the dataset.Availabilitygithub.com/Cobanoglu-Lab/h-tSNE

Download Full-text

A novel metric reveals previously unrecognized distortion in dimensionality reduction of scRNA-Seq data

10.1101/689851 ◽

2019 ◽

Cited By ~ 9

Author(s):

Shamus M. Cooley ◽

Timothy Hamilton ◽

J. Christian J. Ray ◽

Eric J. Deeds

Keyword(s):

Dimensionality Reduction ◽

Single Cells ◽

High Dimensional Data ◽

Three Dimensional ◽

Principal Component ◽

Cell Types ◽

High Dimensional ◽

Before And After ◽

Downstream Analysis ◽

High Level

AbstractHigh-dimensional data are becoming increasingly common in nearly all areas of science. Developing approaches to analyze these data and understand their meaning is a pressing issue. This is particularly true for the rapidly growing field of single-cell RNA-Seq (scRNA-Seq), a technique that simultaneously measures the expression of tens of thousands of genes in thousands to millions of single cells. The emerging consensus for analysis workflows reduces the dimensionality of the dataset before performing downstream analysis, such as assignment of cell types. One problem with this approach is that dimensionality reduction can introduce substantial distortion into the data; consider the familiar example of trying to represent the three-dimensional earth as a two-dimensional map. It is currently unclear if such distortion affects analysis of scRNA-Seq data sets. Here, we introduce a straightforward approach to quantifying this distortion by comparing the local neighborhoods of points before and after dimensionality reduction. We found that popular techniques like t-SNE and UMAP introduce substantial distortion even for relatively simple geometries such as simulated hyperspheres. For scRNA-Seq data, we found the distortion in local neighborhoods was often greater than 95% in the representations typically used for downstream analysis. This high level of distortion can readily introduce important errors into cell type identification, pseudotime ordering, and other analyses that rely on local relationships. We found that principal component analysis can generate accurate embeddings of the data, but only when using dimensionalities that are much higher than typically used in scRNA-Seq analysis. We suggest approaches to take these findings into account and call for a new generation of dimensional reduction algorithms that can accurately embed high dimensional data in its true latent dimension.

Download Full-text