Literature Review of Deep Network Compression

Deep networks often possess a vast number of parameters, and their significant redundancy in parameterization has become a widely-recognized property. This presents significant challenges and restricts many deep learning applications, making the focus on reducing the complexity of models while maintaining their powerful performance. In this paper, we present an overview of popular methods and review recent works on compressing and accelerating deep neural networks. We consider not only pruning methods but also quantization methods, and low-rank factorization methods. This review also intends to clarify these major concepts, and highlights their characteristics, advantages, and shortcomings.

Factorization method for solving nonlocal boundary value problems in Banach space

This article deals with the factorization and solution of nonlocal boundary value problems in a Banach space of the abstract form B1u = Au − SΦ(u) − GΨ(A0u) = f, u ∈ D(B1),where A, A0 are linear abstract operators, S, G are vectors of functions, Φ, Ψ are vectors of linear bounded functionals, and u, f are functions. It is shown that the operator B1 under certain conditions can be factorized into a product of two simpler lower order operators as B1 = BB0. Then the solvability and the unique solution of the equation B1u = f easily follow from the solvability conditions and the unique solutions of the equations Bv = f and B0u = v. The universal technique proposed here is essentially different from other factorization methods in the respect that it involves decomposition of both the equation and boundary conditions and delivers the solution in closed form. The method is implemented to solve ordinary and partial Fredholm integro-differential equations.

CSDVP: Compressed Sensing for Drug-Virus Prediction

The 2019 Coronavirus (COVID-19) epidemic has recently hit most countries hard. Therefore, many researchers around the world are looking for a way to control this virus. Examining existing medications and using them to prevent this epidemic can be helpful. Drug repositioning solutions can be effective because designing and discovering a drug can be very time-consuming. Although no drug has been definitively approved for the treatment of this disease, the effectiveness of a few drugs for the treatment of the disease has been observed. In this study, with the help of computational matrix factorization methods, the associations between drugs and viruses have been predicted. By combining the similarities between the drugs and the similarities between the viruses and using the compressed sensing technique, we investigated the association between the drug and the virus. The Compressed Sensing approach to Drug-Virus Prediction (CSDVP) can work well. We compared the proposed method with other methods in this field and found its accuracy is more desirable than other methods. In fact, the CSDVP approach with the Human drug virus database (HDVD) and evaluation through 5-fold CV, with AUC = 0.87 and AUPR = 0.37, can identify the relationship between drugs and viruses. We also investigated the effect of drug properties on model performance improvement using autoencoder. Thus, with each decrease in the size of the characteristics in different sizes, we examined the performance of the CSDVP model in predicting the drug-virus relationship. The relationship between drugs and coronavirus infection is also analyzed, and the results are presented.

CSDVP: Compressed Sensing for Drug-Virus Prediction

The 2019 Coronavirus (COVID-19) epidemic has recently hit most countries hard. Therefore, many researchers around the world are looking for a way to control this virus. Examining existing medications and using them to prevent this epidemic can be helpful. Drug repositioning solutions can be effective because designing and discovering a drug can be very time-consuming. Although no drug has been definitively approved for the treatment of this disease, the effectiveness of a few drugs for the treatment of the disease has been observed. In this study, with the help of computational matrix factorization methods, the associations between drugs and viruses have been predicted. By combining the similarities between the drugs and the similarities between the viruses and using the compressed sensing technique, we investigated the association between the drug and the virus. The Compressed Sensing approach to Drug-Virus Prediction (CSDVP) can work well. We compared the proposed method with other methods in this field and found its accuracy is more desirable than other methods. In fact, the CSDVP approach with the HDVD dataset and evaluation through 5-fold CV, with AUC = 0.96 and AUPR = 0.85, can identify the relationship between drugs and viruses. We also investigated the effect of drug properties on model performance improvement using autoencoder. Thus, with each decrease in the size of the characteristics in different sizes, we examined the performance of the CSDVP model in predicting the drug-virus relationship. The relationship between drugs and coronavirus infection is also analyzed, and the results are presented.

A Multiple Comprehensive Analysis of scATAC-seq Based on Auto-Encoder and Matrix Decomposition

Single-cell ATAC-seq (scATAC-seq), as the updating of ATAC-seq, provides a novel method for probing open chromatin sites. Currently, research of scATAC-seq is faced with the problem of high dimensionality and the inherent sparsity of the generated data. Recently, several works proposed the use of an autoencoder–decoder, a symmetry neural network architecture, and non-negative matrix factorization methods to characterize the high-dimensional data. To evaluate the performance of multiple methods, in this work, we performed a multiple comparison for characterizing scATAC-seq based on four kinds of auto-encoders known as a symmetry neural network, and two kinds of matrix factorization methods. Different sizes of latent features were used to generate the UMAP plots and for further K-means clustering. Using a gold-standard data set, we practically explored the performance among the methods and the number of latent features in a comprehensive way. Finally, we briefly discuss the underlying difficulties and future directions for scATAC-seq characterizing. As a result, the method designed for handling the sparsity outperforms other tools in the generated dataset.

Missing value imputation on multidimensional time series

We present DeepMVI, a deep learning method for missing value imputation in multidimensional time-series datasets. Missing values are commonplace in decision support platforms that aggregate data over long time stretches from disparate sources, whereas reliable data analytics calls for careful handling of missing data. One strategy is imputing the missing values, and a wide variety of algorithms exist spanning simple interpolation, matrix factorization methods like SVD, statistical models like Kalman filters, and recent deep learning methods. We show that often these provide worse results on aggregate analytics compared to just excluding the missing data. DeepMVI expresses the distribution of each missing value conditioned on coarse and fine-grained signals along a time series, and signals from correlated series at the same time. Instead of resorting to linearity assumptions of conventional matrix factorization methods, DeepMVI harnesses a flexible deep network to extract and combine these signals in an end-to-end manner. To prevent over-fitting with high-capacity neural networks, we design a robust parameter training with labeled data created using synthetic missing blocks around available indices. Our neural network uses a modular design with a novel temporal transformer with convolutional features, and kernel regression with learned embeddings. Experiments across ten real datasets, five different missing scenarios, comparing seven conventional and three deep learning methods show that DeepMVI is significantly more accurate, reducing error by more than 50% in more than half the cases, compared to the best existing method. Although slower than simpler matrix factorization methods, we justify the increased time overheads by showing that DeepMVI provides significantly more accurate imputation that finally impacts quality of downstream analytics.

Robust semi-supervised non-negative matrix factorization for binary subspace learning

Non-negative matrix factorization and its extensions were applied to various areas (i.e., dimensionality reduction, clustering, etc.). When the original data are corrupted by outliers and noise, most of non-negative matrix factorization methods cannot achieve robust factorization and learn a subspace with binary codes. This paper puts forward a robust semi-supervised non-negative matrix factorization method for binary subspace learning, called RSNMF, for image clustering. For better clustering performance on the dataset contaminated by outliers and noise, we propose a weighted constraint on the noise matrix and impose manifold learning into non-negative matrix factorization. Moreover, we utilize the discrete hashing learning method to constrain the learned subspace, which can achieve a binary subspace from the original data. Experimental results validate the robustness and effectiveness of RSNMF in binary subspace learning and image clustering on the face dataset corrupted by Salt and Pepper noise and Contiguous Occlusion.

The Power of Matrix Factorization: Methods for Deconvoluting Genetic Heterogeneous Data at Expression Level

Background:: The utilization of genetic data to investigate biological problems has recently become a vital approach. However, it is undeniable that the heterogeneity of original samples at the biological level is usually ignored when utilizing genetic data. Different cell-constitutions of a sample could differentiate the expression profile, and set considerable biases for downstream research. Matrix factorization (MF) which originated as a set of mathematical methods, has contributed massively to deconvoluting genetic profiles in silico, especially at the expression level. Objective: With the development of artificial intelligence algorithms and machine learning, the number of computational methods for solving heterogeneous problems is also rapidly abundant. However, a structural view from the angle of using MF to deconvolute genetic data is quite limited. This study was conducted to review the usages of MF methods on heterogeneous problems of genetic data on expression level. Methods: MF methods involved in deconvolution were reviewed according to their individual strengths. The demonstration is presented separately into three sections: application scenarios, method categories and summarization for tools. Specifically, application scenarios defined deconvoluting problem with applying scenarios. Method categories summarized MF algorithms contributed to different scenarios. Summarization for tools listed functions and developed web-servers over the latest decade. Additionally, challenges and opportunities of relative fields are discussed. Results and Conclusion: Based on the investigation, this study aims to present a relatively global picture to assist researchers to achieve a quicker access of deconvoluting genetic data in silico, further to help researchers in selecting suitable MF methods based on the different scenarios.