scholarly journals Z-checker: A framework for assessing lossy compression of scientific data

2017 ◽  
Vol 33 (2) ◽  
pp. 285-303 ◽  
Author(s):  
Dingwen Tao ◽  
Sheng Di ◽  
Hanqi Guo ◽  
Zizhong Chen ◽  
Franck Cappello
Keyword(s):  
Mean Squared Error ◽  
Signal To Noise Ratio ◽  
Rate Distortion ◽  
Principal Component ◽  
Original Data ◽  
Lossy Compression ◽  
Scientific Data ◽  
Average Error ◽  
Data Sets ◽  
Data Set

Because of the vast volume of data being produced by today’s scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific data sets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this article, we present a survey of existing lossy compressors. Then, we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties (such as entropy, distribution, power spectrum, principal component analysis, and autocorrelation) of any data set to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality (compression ratio and bit rate) and provide various global distortion analysis comparing the original data with the decompressed data (peak signal-to-noise ratio, normalized mean squared error, rate–distortion, rate-compression error, spectral, distribution, and derivatives) and statistical analysis of the compression error (maximum, minimum, and average error; autocorrelation; and distribution of errors). Z-checker can perform the analysis with either coarse granularity (throughout the whole data set) or fine granularity (by user-defined blocks), such that the users and developers can select the best fit, adaptive compressors for different parts of the data set. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the data sets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific data sets.

scholarly journals Prewhitening High-Dimensional fMRI Data Sets Without Eigendecomposition

2014 ◽  
Vol 26 (5) ◽  
pp. 907-919 ◽  
Author(s):  
Abd-Krim Seghouane ◽  
Yousef Saad
Keyword(s):  
Covariance Matrix ◽  
Mean Squared Error ◽  
Computational Cost ◽  
Original Data ◽  
Fmri Data ◽  
High Dimensional ◽  
Data Sets ◽  
Data Set ◽  
Low Computational Cost ◽  
Eigenvectors And Eigenvalues

This letter proposes an algorithm for linear whitening that minimizes the mean squared error between the original and whitened data without using the truncated eigendecomposition (ED) of the covariance matrix of the original data. This algorithm uses Lanczos vectors to accurately approximate the major eigenvectors and eigenvalues of the covariance matrix of the original data. The major advantage of the proposed whitening approach is its low computational cost when compared with that of the truncated ED. This gain comes without sacrificing accuracy, as illustrated with an experiment of whitening a high-dimensional fMRI data set.

scholarly journals Optimal distribution-preserving downsampling of large biomedical data sets (opdisDownsampling)

PLoS ONE ◽  
2021 ◽  
Vol 16 (8) ◽  
pp. e0255838
Author(s):  
Jörn Lötsch ◽  
Sebastian Malkusch ◽  
Alfred Ultsch
Keyword(s):  
Standard Method ◽  
Selection Criterion ◽  
Principal Component ◽  
Original Data ◽  
Data Sets ◽  
Biomedical Data ◽  
Optimal Distribution ◽  
Current Standard ◽  
Data Set ◽  
Entire Data

Motivation The size of today’s biomedical data sets pushes computer equipment to its limits, even for seemingly standard analysis tasks such as data projection or clustering. Reducing large biomedical data by downsampling is therefore a common early step in data processing, often performed as random uniform class-proportional downsampling. In this report, we hypothesized that this can be optimized to obtain samples that better reflect the entire data set than those obtained using the current standard method. Results By repeating the random sampling and comparing the distribution of the drawn sample with the distribution of the original data, it was possible to establish a method for obtaining subsets of data that better reflect the entire data set than taking only the first randomly selected subsample, as is the current standard. Experiments on artificial and real biomedical data sets showed that the reconstruction of the remaining data from the original data set from the downsampled data improved significantly. This was observed with both principal component analysis and autoencoding neural networks. The fidelity was dependent on both the number of cases drawn from the original and the number of samples drawn. Conclusions Optimal distribution-preserving class-proportional downsampling yields data subsets that reflect the structure of the entire data better than those obtained with the standard method. By using distributional similarity as the only selection criterion, the proposed method does not in any way affect the results of a later planned analysis.

Application of multivariable statistical techniques in plant-wide WWTP control strategies analysis

2007 ◽  
Vol 56 (6) ◽  
pp. 75-83 ◽  
Author(s):  
X. Flores ◽  
J. Comas ◽  
I.R. Roda ◽  
L. Jiménez ◽  
K.V. Gernaey
Keyword(s):  
Cluster Analysis ◽  
Control Strategies ◽  
Treatment Plant ◽  
Principal Component ◽  
Statistical Techniques ◽  
Data Sets ◽  
Data Set ◽  
Casual Relation ◽  
Evaluation Matrix ◽  
Natural Groups

The main objective of this paper is to present the application of selected multivariable statistical techniques in plant-wide wastewater treatment plant (WWTP) control strategies analysis. In this study, cluster analysis (CA), principal component analysis/factor analysis (PCA/FA) and discriminant analysis (DA) are applied to the evaluation matrix data set obtained by simulation of several control strategies applied to the plant-wide IWA Benchmark Simulation Model No 2 (BSM2). These techniques allow i) to determine natural groups or clusters of control strategies with a similar behaviour, ii) to find and interpret hidden, complex and casual relation features in the data set and iii) to identify important discriminant variables within the groups found by the cluster analysis. This study illustrates the usefulness of multivariable statistical techniques for both analysis and interpretation of the complex multicriteria data sets and allows an improved use of information for effective evaluation of control strategies.

Bayesian Classifier for Sparsity-Promoting Feature Selection

2015 ◽  
Vol 29 (06) ◽  
pp. 1550022 ◽  
Author(s):  
Danlei Xu ◽  
Lan Du ◽  
Hongwei Liu ◽  
Penghui Wang
Keyword(s):  
Feature Selection ◽  
Synthetic Data ◽  
Original Data ◽  
Radar Data ◽  
Bayesian Classifier ◽  
Classification Model ◽  
Data Sets ◽  
Data Set ◽  
Classification Boundary ◽  
Nonlinear Mappings

A Bayesian classifier for sparsity-promoting feature selection is developed in this paper, where a set of nonlinear mappings for the original data is performed as a pre-processing step. The linear classification model with such mappings from the original input space to a nonlinear transformation space can not only construct the nonlinear classification boundary, but also realize the feature selection for the original data. A zero-mean Gaussian prior with Gamma precision and a finite approximation of Beta process prior are used to promote sparsity in the utilization of features and nonlinear mappings in our model, respectively. We derive the Variational Bayesian (VB) inference algorithm for the proposed linear classifier. Experimental results based on the synthetic data set, measured radar data set, high-dimensional gene expression data set, and several benchmark data sets demonstrate the aggressive and robust feature selection capability and comparable classification accuracy of our method comparing with some other existing classifiers.

Dimensionality and Its Reduction

2014 ◽  
Author(s):  
Andrew J. Connolly ◽  
Jacob T. VanderPlas ◽  
Alexander Gray ◽  
Andrew J. Connolly ◽  
Jacob T. VanderPlas ◽  
...  
Keyword(s):  
Principal Component Analysis ◽  
Principal Component ◽  
Reduction Technique ◽  
High Dimensional ◽  
Data Sets ◽  
Data Set ◽  
Gaussian Distributions ◽  
Dimensionality Reduction Technique ◽  
Alternative Techniques ◽  
New Generation

With the dramatic increase in data available from a new generation of astronomical telescopes and instruments, many analyses must address the question of the complexity as well as size of the data set. This chapter deals with how we can learn which measurements, properties, or combinations thereof carry the most information within a data set. It describes techniques that are related to concepts discussed when describing Gaussian distributions, density estimation, and the concepts of information content. The chapter begins with an exploration of the problems posed by high-dimensional data. It then describes the data sets used in this chapter, and introduces perhaps the most important and widely used dimensionality reduction technique, principal component analysis (PCA). The remainder of the chapter discusses several alternative techniques which address some of the weaknesses of PCA.

Making more research count: a blockchain enabled one-stop shop for immutable behavioral research

foresight ◽  
2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Christian Hugo Hoffmann
Keyword(s):  
Institutional Economics ◽  
New Technologies ◽  
Behavioral Science ◽  
Data Integrity ◽  
Original Data ◽  
Scientific Data ◽  
Data Set ◽  
Content Type ◽  
Blockchain Technology ◽  
The Right

Purpose The purpose of this paper is to offer a panoramic view at the credibility issues that exist within social sciences research. Design/methodology/approach The central argument of this paper is that a joint effort between blockchain and other technologies such as artificial intelligence (AI) and deep learning and how they can prevent scientific data manipulation or data forgery as a way to make science more decentralized and anti-fragile, without losing data integrity or reputation as a trade-off. The authors address it by proposing an online research platform for use in social and behavioral science that guarantees data integrity through a combination of modern institutional economics and blockchain technology. Findings The benefits are mainly twofold: On the one hand, social science scholars get paired with the right target audience for their studies. On the other hand, a snapshot of the gathered data at the time of creation is taken so that researchers can prove that they used the original data set to peers in the future while maintaining full control of their data. Originality/value The proposed combination of behavioral economics with new technologies such as blockchain and AI is novel and translated into a cutting-edge tool to be implemented.

Feature-Based Uncertainty Visualization

Big Data ◽  
2016 ◽  
pp. 261-287
Author(s):  
Keqin Wu ◽  
Song Zhang
Keyword(s):  
Scientific Data ◽  
Data Sets ◽  
Data Set ◽  
Uncertainty Visualization ◽  
Scalar Data ◽  
Critical Issues ◽  
Contour Level ◽  
Feature Based ◽  
2D Data ◽  
The Impact

While uncertainty in scientific data attracts an increasing research interest in the visualization community, two critical issues remain insufficiently studied: (1) visualizing the impact of the uncertainty of a data set on its features and (2) interactively exploring 3D or large 2D data sets with uncertainties. In this chapter, a suite of feature-based techniques is developed to address these issues. First, an interactive visualization tool for exploring scalar data with data-level, contour-level, and topology-level uncertainties is developed. Second, a framework of visualizing feature-level uncertainty is proposed to study the uncertain feature deviations in both scalar and vector data sets. With quantified representation and interactive capability, the proposed feature-based visualizations provide new insights into the uncertainties of both data and their features which otherwise would remain unknown with the visualization of only data uncertainties.

scholarly journals Decomposing the Apoptosis Pathway Into Biologically Interpretable Principal Components

2018 ◽  
Vol 17 ◽  
pp. 117693511877108 ◽  
Author(s):  
Min Wang ◽  
Steven M Kornblau ◽  
Kevin R Coombes
Keyword(s):  
Principal Components ◽  
Myeloid Leukemia ◽  
Principal Component ◽  
R Package ◽  
Biological Data ◽  
Data Sets ◽  
Proteomics Data ◽  
Data Set ◽  
Apoptosis Pathway ◽  
Biological Interpretation

Principal component analysis (PCA) is one of the most common techniques in the analysis of biological data sets, but applying PCA raises 2 challenges. First, one must determine the number of significant principal components (PCs). Second, because each PC is a linear combination of genes, it rarely has a biological interpretation. Existing methods to determine the number of PCs are either subjective or computationally extensive. We review several methods and describe a new R package, PCDimension, that implements additional methods, the most important being an algorithm that extends and automates a graphical Bayesian method. Using simulations, we compared the methods. Our newly automated procedure is competitive with the best methods when considering both accuracy and speed and is the most accurate when the number of objects is small compared with the number of attributes. We applied the method to a proteomics data set from patients with acute myeloid leukemia. Proteins in the apoptosis pathway could be explained using 6 PCs. By clustering the proteins in PC space, we were able to replace the PCs by 6 “biological components,” 3 of which could be immediately interpreted from the current literature. We expect this approach combining PCA with clustering to be widely applicable.

scholarly journals Dynamic spatio-temporal generation of large-scale synthetic gridded precipitation: with improved spatial coherence of extremes

2019 ◽  
Vol 34 (9) ◽  
pp. 1369-1383 ◽  
Author(s):  
Dirk Diederen ◽  
Ye Liu
Keyword(s):  
Large Scale ◽  
Spatial Coherence ◽  
Original Data ◽  
Return Level ◽  
Data Sets ◽  
Large Set ◽  
Precipitation Data ◽  
Data Set ◽  
Spatio Temporal ◽  
Synthetic Precipitation

Abstract With the ongoing development of distributed hydrological models, flood risk analysis calls for synthetic, gridded precipitation data sets. The availability of large, coherent, gridded re-analysis data sets in combination with the increase in computational power, accommodates the development of new methodology to generate such synthetic data. We tracked moving precipitation fields and classified them using self-organising maps. For each class, we fitted a multivariate mixture model and generated a large set of synthetic, coherent descriptors, which we used to reconstruct moving synthetic precipitation fields. We introduced randomness in the original data set by replacing the observed precipitation fields in the original data set with the synthetic precipitation fields. The output is a continuous, gridded, hourly precipitation data set of a much longer duration, containing physically plausible and spatio-temporally coherent precipitation events. The proposed methodology implicitly provides an important improvement in the spatial coherence of precipitation extremes. We investigate the issue of unrealistic, sudden changes on the grid and demonstrate how a dynamic spatio-temporal generator can provide spatial smoothness in the probability distribution parameters and hence in the return level estimates.

scholarly journals So you think you can PLS-DA?

2020 ◽  
Vol 21 (S1) ◽  
Author(s):  
Daniel Ruiz-Perez ◽  
Haibin Guan ◽  
Purnima Madhivanan ◽  
Kalai Mathee ◽  
Giri Narasimhan
Keyword(s):  
Feature Selection ◽  
Signal To Noise Ratio ◽  
Synthetic Data ◽  
Principal Component ◽  
Ground Truth ◽  
Close Relative ◽  
Data Set ◽  
Series Of Experiments ◽  
Feature Selector ◽  
Class Labels

Abstract Background Partial Least-Squares Discriminant Analysis (PLS-DA) is a popular machine learning tool that is gaining increasing attention as a useful feature selector and classifier. In an effort to understand its strengths and weaknesses, we performed a series of experiments with synthetic data and compared its performance to its close relative from which it was initially invented, namely Principal Component Analysis (PCA). Results We demonstrate that even though PCA ignores the information regarding the class labels of the samples, this unsupervised tool can be remarkably effective as a feature selector. In some cases, it outperforms PLS-DA, which is made aware of the class labels in its input. Our experiments range from looking at the signal-to-noise ratio in the feature selection task, to considering many practical distributions and models encountered when analyzing bioinformatics and clinical data. Other methods were also evaluated. Finally, we analyzed an interesting data set from 396 vaginal microbiome samples where the ground truth for the feature selection was available. All the 3D figures shown in this paper as well as the supplementary ones can be viewed interactively at http://biorg.cs.fiu.edu/plsda Conclusions Our results highlighted the strengths and weaknesses of PLS-DA in comparison with PCA for different underlying data models.

Sign in / Sign up

Export Citation Format

Share Document