scholarly journals Genomic data imputation with variational auto-encoders

GigaScience ◽  
2020 ◽  
Vol 9 (8) ◽  
Author(s):  
Yeping Lina Qiu ◽  
Hong Zheng ◽  
Olivier Gevaert
Keyword(s):  
Deep Learning ◽  
Missing Data ◽  
Large Scale ◽  
Missing Values ◽  
Genomic Data ◽  
Large Data ◽  
Missing At Random ◽  
Data Sets ◽  
Data Imputation ◽  
Missing Not At Random

Abstract Background As missing values are frequently present in genomic data, practical methods to handle missing data are necessary for downstream analyses that require complete data sets. State-of-the-art imputation techniques, including methods based on singular value decomposition and K-nearest neighbors, can be computationally expensive for large data sets and it is difficult to modify these algorithms to handle certain cases not missing at random. Results In this work, we use a deep-learning framework based on the variational auto-encoder (VAE) for genomic missing value imputation and demonstrate its effectiveness in transcriptome and methylome data analysis. We show that in the vast majority of our testing scenarios, VAE achieves similar or better performances than the most widely used imputation standards, while having a computational advantage at evaluation time. When dealing with data missing not at random (e.g., few values are missing), we develop simple yet effective methodologies to leverage the prior knowledge about missing data. Furthermore, we investigate the effect of varying latent space regularization strength in VAE on the imputation performances and, in this context, show why VAE has a better imputation capacity compared to a regular deterministic auto-encoder. Conclusions We describe a deep learning imputation framework for transcriptome and methylome data using a VAE and show that it can be a preferable alternative to traditional methods for data imputation, especially in the setting of large-scale data and certain missing-not-at-random scenarios.

scholarly journals Kernel weighted least square approach for imputing missing values of metabolomics data

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Nishith Kumar ◽  
Md. Aminul Hoque ◽  
Masahiro Sugimoto
Keyword(s):  
Missing Data ◽  
Large Scale ◽  
Missing Values ◽  
Kernel Weight ◽  
Least Square ◽  
Data Matrix ◽  
Data Imputation ◽  
Metabolomics Data ◽  
Missing Value ◽  
Missing Data Imputation

AbstractMass spectrometry is a modern and sophisticated high-throughput analytical technique that enables large-scale metabolomic analyses. It yields a high-dimensional large-scale matrix (samples × metabolites) of quantified data that often contain missing cells in the data matrix as well as outliers that originate for several reasons, including technical and biological sources. Although several missing data imputation techniques are described in the literature, all conventional existing techniques only solve the missing value problems. They do not relieve the problems of outliers. Therefore, outliers in the dataset decrease the accuracy of the imputation. We developed a new kernel weight function-based proposed missing data imputation technique that resolves the problems of missing values and outliers. We evaluated the performance of the proposed method and other conventional and recently developed missing imputation techniques using both artificially generated data and experimentally measured data analysis in both the absence and presence of different rates of outliers. Performances based on both artificial data and real metabolomics data indicate the superiority of our proposed kernel weight-based missing data imputation technique to the existing alternatives. For user convenience, an R package of the proposed kernel weight-based missing value imputation technique was developed, which is available at https://github.com/NishithPaul/tWLSA.

scholarly journals A deep learning framework for imputing missing values in genomic data

2018 ◽  
Author(s):  
Yeping Lina Qiu ◽  
Hong Zheng ◽  
Olivier Gevaert
Keyword(s):  
Deep Learning ◽  
Missing Values ◽  
Gc Content ◽  
Genomic Data ◽  
Missing At Random ◽  
Denoising Autoencoder ◽  
K Nearest Neighbors ◽  
Learning Framework ◽  
Value Decomposition ◽  
Pan Cancer

AbstractMotivationThe presence of missing values is a frequent problem encountered in genomic data analysis. Lost data can be an obstacle to downstream analyses that require complete data matrices. State-of-the-art imputation techniques including Singular Value Decomposition (SVD) and K-Nearest Neighbors (KNN) based methods usually achieve good performances, but are computationally expensive especially for large datasets such as those involved in pan-cancer analysis.ResultsThis study describes a new method: a denoising autoencoder with partial loss (DAPL) as a deep learning based alternative for data imputation. Results on pan-cancer gene expression data and DNA methylation data from over 11,000 samples demonstrate significant improvement over standard denoising autoencoder for both data missing-at-random cases with a range of missing percentages, and missing-not-at-random cases based on expression level and GC-content. We discuss the advantages of DAPL over traditional imputation methods and show that it achieves comparable or better performance with less computational burden.Availabilityhttps://github.com/gevaertlab/[email protected]

scholarly journals Machine learning based imputation techniques for estimating phylogenetic trees from incomplete distance matrices

2019 ◽  
Author(s):  
Ananya Bhattacharjee ◽  
Md. Shamsuzzoha Bayzid
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Missing Data ◽  
Phylogenetic Trees ◽  
Large Scale ◽  
Missing Values ◽  
Gene Tree ◽  
Estimation Methods ◽  
Learning Technique ◽  
Distance Matrices

AbstractBackgroundDue to the recent advances in sequencing technologies and species tree estimation methods capable of taking gene tree discordance into account, notable progress has been achieved in constructing large scale phylogenetic trees from genome wide data. However, substantial challenges remain in leveraging this huge amount of molecular data. One of the foremost among these challenges is the need for efficient tools that can handle missing data. Popular distance-based methods such as neighbor joining and UPGMA require that the input distance matrix does not contain any missing values.ResultsWe introduce two highly accurate machine learning based distance imputation techniques. One of our approaches is based on matrix factorization, and the other one is an autoencoder based deep learning technique. We evaluate these two techniques on a collection of simulated and biological datasets, and show that our techniques match or improve upon the best alternate techniques for distance imputation. Moreover, our proposed techniques can handle substantial amount of missing data, to the extent where the best alternate methods fail.ConclusionsThis study shows for the first time the power and feasibility of applying deep learning techniques for imputing distance matrices. The autoencoder based deep learning technique is highly accurate and scalable to large dataset. We have made these techniques freely available as a cross-platform software (available at https://github.com/Ananya-Bhattacharjee/ImputeDistances).

scholarly journals Kernel Weighted Least Square Approach for Imputing Missing Values of Metabolomics Data

2021 ◽  
Author(s):  
Nishith Kumar ◽  
Md. Hoque ◽  
Masahiro Sugimoto
Keyword(s):  
Missing Data ◽  
Large Scale ◽  
Missing Values ◽  
Kernel Weight ◽  
Least Square ◽  
Data Matrix ◽  
Data Imputation ◽  
Metabolomics Data ◽  
Missing Value ◽  
Missing Data Imputation

Abstract Mass spectrometry is a modern and sophisticated high-throughput analytical technique that enables large-scale metabolomics analyses. It yields a high dimensional large scale matrix (samples × metabolites) of quantified data that often contain missing cell in the data matrix as well as outliers which originate from several reasons, including technical and biological sources. Although, in the literature, several missing data imputation techniques can be found, however all the conventional existing techniques can only solve the missing value problems but not relieve the problems of outliers. Therefore, outliers in the dataset, deteriorate the accuracy of imputation. To overcome both the missing data imputation and outlier’s problem, here, we developed a new kernel weight function based missing data imputation technique (proposed) that resolves both the missing values and outliers. We evaluated the performance of the proposed method and other nine conventional missing imputation techniques using both artificially generated data and experimentally measured data analysis in both absence and presence of different rates of outliers. Performance based on both artificial data and real metabolomics data indicates that our proposed kernel weight based missing data imputation technique is a better performer than some existing alternatives. For user convenience, an R package of the proposed kernel weight based missing value imputation technique has been developed which is available at https://github.com/NishithPaul/tWLSA .

A large-scale sensor missing data imputation framework for dams using deep learning and transfer learning strategy

Measurement ◽  
2021 ◽  
pp. 109377
Author(s):  
Yangtao Li ◽  
Tengfei Bao ◽  
Hao Chen ◽  
Kang Zhang Data analysis ◽  
Xiaosong Shu ◽  
...  
Keyword(s):  
Deep Learning ◽  
Missing Data ◽  
Transfer Learning ◽  
Large Scale ◽  
Learning Strategy ◽  
Data Imputation ◽  
Missing Data Imputation

scholarly journals Mind the Gap: The Prospects of Missing Data

2016 ◽  
Vol 8 (5) ◽  
pp. 708-712 ◽  
Author(s):  
Meghan McConnell ◽  
Jonathan Sherbino ◽  
Teresa M. Chan
Keyword(s):  
Missing Data ◽  
Large Data ◽  
Missing At Random ◽  
Large Data Sets ◽  
Data Sets ◽  
Resident Performance ◽  
Rank Score ◽  
Data Points ◽  
Health Advocate ◽  
Competency Based

ABSTRACT Background  The increasing use of workplace-based assessments (WBAs) in competency-based medical education has led to large data sets that assess resident performance longitudinally. With large data sets, problems that arise from missing data are increasingly likely. Objective  The purpose of this study is to examine (1) whether data are missing at random across various WBAs, and (2) the relationship between resident performance and the proportion of missing data. Methods  During 2012–2013, a total of 844 WBAs of CanMEDs Roles were completed for 9 second-year emergency medicine residents. To identify whether missing data were randomly distributed across various WBAs, the total number of missing data points was calculated for each Role. To examine whether the amount of missing data was related to resident performance, 5 faculty members rank-ordered the residents based on performance. A median rank score was calculated for each resident and was correlated with the proportion of missing data. Results  More data were missing for Health Advocate and Professional WBAs relative to other competencies (P < .001). Furthermore, resident rankings were not related to the proportion of missing data points (r = 0.29, P > .05). Conclusions  The results of the present study illustrate that some CanMEDS Roles are less likely to be assessed than others. At the same time, the amount of missing data did not correlate with resident performance, suggesting lower-performing residents are no more likely to have missing data than their higher-performing peers. This article discusses several approaches to dealing with missing data.

IMPUTATION OF MISSING DATA WITH DIFFERENT MISSINGNESS MECHANISM

2012 ◽  
Vol 57 (1) ◽  
Author(s):  
HO MING KANG ◽  
FADHILAH YUSOF ◽  
ISMAIL MOHAMAD
Keyword(s):  
Missing Data ◽  
Missing Values ◽  
Missing At Random ◽  
Absolute Error ◽  
Data Sets ◽  
Missing Completely At Random ◽  
Missingness Mechanism ◽  
Mean Imputation ◽  
The Mean ◽  
Estimation Of Missing Data

This paper presents a study on the estimation of missing data. Data samples with different missingness mechanism namely Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) are simulated accordingly. Expectation maximization (EM) algorithm and mean imputation (MI) are applied to these data sets and compared and the performances are evaluated by the mean absolute error (MAE) and root mean square error (RMSE). The results showed that EM is able to estimate the missing data with minimum errors compared to mean imputation (MI) for the three missingness mechanisms. However the graphical results showed that EM failed to estimate the missing values in the missing quadrants when the situation is MNAR.

scholarly journals Improving accuracy of missing data imputation in data mining

2017 ◽  
Vol 2 (3) ◽  
pp. 66-73
Author(s):  
Nzar A. Ali ◽  
Zhyan M. Omer
Keyword(s):  
Data Mining ◽  
Missing Data ◽  
Real World ◽  
Missing Values ◽  
Large Data ◽  
Data Repository ◽  
Data Sets ◽  
Real World Data ◽  
Missing Data Imputation ◽  
Improving Accuracy

In fact, raw data in the real world is dirty. Each large data repository contains various types of anomalous values that influence the result of the analysis, since in data mining, good models usually need good data, databases in the world are not always clean and includes noise, incomplete data, duplicate records, inconsistent data and missing values. Missing data is a common drawback in many real-world data sets. In this paper, we proposed an algorithm depending on improving (MIGEC) algorithm in the way of imputation for dealing missing values. We implement grey relational analysis (GRA) on attribute values instead of instance values, and the missing data were initially imputed by mean imputation and then estimated by our proposed algorithm (PA) used as a complete value for imputing next missing value.We compare our proposed algorithm with several other algorithms such as MMS, HDI, KNNMI, FCMOCS, CRI, CMI, NIIA and MIGEC under different missing mechanisms. Experimental results demonstrate that the proposed algorithm has less RMSE values than other algorithms under all missingness mechanisms.

The Issue of Missing Values in Data Mining

2011 ◽  
pp. 1102-1109
Author(s):  
Malcolm J. Beynon
Keyword(s):  
Data Mining ◽  
Missing Data ◽  
Incomplete Data ◽  
Missing Values ◽  
Large Data ◽  
Original Data ◽  
Customer Relationship ◽  
Data Sets ◽  
Data Mining Technique ◽  
Data Set

The essence of data mining is to investigate for pertinent information that may exist in data (often large data sets). The immeasurably large amount of data present in the world, due to the increasing capacity of storage media, manifests the issue of the presence of missing values (Olinsky et al., 2003; Brown and Kros, 2003). The presented encyclopaedia article considers the general issue of the presence of missing values when data mining, and demonstrates the effect of when managing their presence is or is not undertaken, through the utilisation of a data mining technique. The issue of missing values was first exposited over forty years ago in Afifi and Elashoff (1966). Since then it is continually the focus of study and explanation (El-Masri and Fox-Wasylyshyn, 2005), covering issues such as the nature of their presence and management (Allison, 2000). With this in mind, the naïve consistent aspect of the missing value debate is the limited general strategies available for their management, the main two being either the simple deletion of cases with missing data or a form of imputation of the missing values in someway (see Elliott and Hawthorne, 2005). Examples of the specific investigation of missing data (and data quality), include in; data warehousing (Ma et al., 2000), and customer relationship management (Berry and Linoff, 2000). An alternative strategy considered is the retention of the missing values, and their subsequent ‘ignorance’ contribution in any data mining undertaken on the associated original incomplete data set. A consequence of this retention is that full interpretability can be placed on the results found from the original incomplete data set. This strategy can be followed when using the nascent CaRBS technique for object classification (Beynon, 2005a, 2005b). CaRBS analyses are presented here to illustrate that data mining can manage the presence of missing values in a much more effective manner than the more inhibitory traditional strategies. An example data set is considered, with a noticeable level of missing values present in the original data set. A critical increase in the number of missing values present in the data set further illustrates the benefit from ‘intelligent’ data mining (in this case using CaRBS).

scholarly journals Galaxy spin direction distribution in HST and SDSS show similar large-scale asymmetry

2020 ◽  
Vol 37 ◽  
Author(s):  
Lior Shamir
Keyword(s):  
Large Scale ◽  
Spiral Galaxies ◽  
Hubble Space Telescope ◽  
Gravitational Interaction ◽  
Large Data ◽  
Sloan Digital Sky Survey ◽  
Data Sets ◽  
Dipole Axis ◽  
Data Set ◽  
The Asymmetry

Abstract Several recent observations using large data sets of galaxies showed non-random distribution of the spin directions of spiral galaxies, even when the galaxies are too far from each other to have gravitational interaction. Here, a data set of $\sim8.7\cdot10^3$ spiral galaxies imaged by Hubble Space Telescope (HST) is used to test and profile a possible asymmetry between galaxy spin directions. The asymmetry between galaxies with opposite spin directions is compared to the asymmetry of galaxies from the Sloan Digital Sky Survey. The two data sets contain different galaxies at different redshift ranges, and each data set was annotated using a different annotation method. The results show that both data sets show a similar asymmetry in the COSMOS field, which is covered by both telescopes. Fitting the asymmetry of the galaxies to cosine dependence shows a dipole axis with probabilities of $\sim2.8\sigma$ and $\sim7.38\sigma$ in HST and SDSS, respectively. The most likely dipole axis identified in the HST galaxies is at $(\alpha=78^{\rm o},\delta=47^{\rm o})$ and is well within the $1\sigma$ error range compared to the location of the most likely dipole axis in the SDSS galaxies with $z>0.15$ , identified at $(\alpha=71^{\rm o},\delta=61^{\rm o})$ .

Sign in / Sign up

Export Citation Format

Share Document