STRETCHING THE NET: MULTIDIMENSIONAL REGULARIZATION

This paper derives asymptotic risk (expected loss) results for shrinkage estimators with multidimensional regularization in high-dimensional settings. We introduce a class of multidimensional shrinkage estimators (MuSEs), which includes the elastic net, and show that—as the number of parameters to estimate grows—the empirical loss converges to the oracle-optimal risk. This result holds when the regularization parameters are estimated empirically via cross-validation or Stein’s unbiased risk estimate. To help guide applied researchers in their choice of estimator, we compare the empirical Bayes risk of the lasso, ridge, and elastic net in a spike and normal setting. Of the three estimators, we find that the elastic net performs best when the data are moderately sparse and the lasso performs best when the data are highly sparse. Our analysis suggests that applied researchers who are unsure about the level of sparsity in their data might benefit from using MuSEs such as the elastic net. We exploit these insights to propose a new estimator, the cubic net, and demonstrate through simulations that it outperforms the three other estimators for any sparsity level.

Dynamic Regularization on Activation Sparsity for Neural Network Efficiency Improvement

When deploying deep neural networks in embedded systems, it is crucial to decrease the model size and computational complexity for improving the execution speed and efficiency. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can also dramatically reduce the amount of data communication and the computation cost. Unlike weight parameters, the pattern of activations is directly related to input data and thereby changes dynamically. To regulate the dynamic activation sparsity (DAS), in this work, we propose a generic low-cost approach based on winners-take-all (WTA) dropout technique. The network enhanced by the proposed WTA dropout, namely DASNet , features structured activation sparsity with an improved sparsity level. Compared to the static feature map pruning methods, DASNets provide better computation cost reduction. The WTA dropout technique can be easily applied in deep neural networks without incurring additional training variables. More importantly, DASNet can be seamlessly integrated with other compression techniques, such as weight pruning and quantization, without compromising accuracy. Our experiments on various networks and datasets present significant runtime speedups with negligible accuracy losses.

A Novel Sparse Data Reconstruction Algorithm for Dynamically Detect and Adjust Signal Sparsity

This paper proposed a novel algorithm which is called the joint step-size matching pursuit algorithm (JsTMP) to solve the issue of calculating the unknown signal sparsity. The proposed algorithm falls into the general category of greedy algorithms. In the process of iteration, this method can adjust the step size and correct the indices of the estimated support that were erroneously selected in a dynamical way. And it uses the dynamical step sizes to increase the estimated sparsity level when the energy of the residual is less than half of that of the measurement vectory. The main innovations include two aspects: 1) The high probability of exact reconstruction, comparable to other classical greedy algorithms reconstruct arbitrary spare signal. 2) The sinh() function is used to adjust the right step with the value of the objective function in the late iteration. Finally, by following this approach, the simulation results show that the proposed algorithm outperforms state of- the-art similar algorithms used for solving the same problem.

Implementation of Sparse Techniques for MIMO Channel Estimation

Channel estimation plays a very important role on the performance of wireless communication systems.Channel estimation can be carried out in different ways: with or without the help of a parametric model, using frequency and/or time correlation properties in the wireless channel, blind methods or those based on training pilots, adaptive or non-adaptive methods. As more antennas are added to Multiple Input Multiple Output (MIMO) system more computational resources and power are required. There is a need to address this problem of the overhead in training symbol based models for channel estimation. Compressed Sensing (CS) algorithms are beneficial in addressing these limitations in the system to increase the spectral efficiency can help free up resources and prevent additional taxing on the hardware. Compressed sensing the pilot symbols by using CS algorithms like OMP, SP, and CoSaMP algorithms. The channel coefficients are obtained through LS, MMSE and LMS techniques. Then, a very small amount of frequency-domain orthogonal pilots are used for the accurate channel estimation.The traditional algorithms like LS , MMSE and LMS combined with CS algorithms have better performance at low SNRs compared to conventional techniques alone in terms of computational time complexity and Normalised Mean Square Error(NMSE) performance. There is a halving of pilot symbols used for training by using the CS algorithm. MSE performance is increased with increase in sparsity level. CoSaMP performs better than SP and OMP at low SNRs. With increase in sparsity level after 50K, the performance of SP is comparable to that of CoSaMP. OMP is simple to implement but MSE performance is less and computational time is more compared to SP and CoSaMP.

Filtering-Based Regularized Sparsity Variable Step-Size Matching Pursuit and Its Applications in Vehicle Health Monitoring

Recent years have witnessed that real-time health monitoring for vehicles is gaining importance. Conventional monitoring scheme faces formidable challenges imposed by the massive signals generated with extremely heavy burden on storage and transmission. To address issues of signal sampling and transmission, compressed sensing (CS) has served as a promising solution in vehicle health monitoring, which performs signal sampling and compression simultaneously. Signal reconstruction is regarded as the most critical part of CS, while greedy reconstruction has been a research hotspot. However, the existing approaches either require prior knowledge of the sparse signal or perform with expensive computational complexity. To exploit the structure of the sparse signal, in this paper, we introduce an initial estimation approach for signal sparsity level firstly. Then, a novel greedy reconstruction algorithm that relies on no prior information of sparsity level while maintaining a good reconstruction performance is presented. The proposed algorithm integrates strategies of regularization and variable adaptive step size and further performs filtration. To verify the efficiency of the algorithm, typical voltage disturbance signals generated by the vehicle power system are taken as trial data. Preliminary simulation results demonstrate that the proposed algorithm achieves superior performance compared to the existing methods.

Sparse covariance estimation in logit mixture models

This paper introduces a new data-driven methodology for estimating sparse covariance matrices of the random coefficients in logit mixture models. Researchers typically specify covariance matrices in logit mixture models under one of two extreme assumptions: either an unrestricted full covariance matrix (allowing correlations between all random coefficients), or a restricted diagonal matrix (allowing no correlations at all). Our objective is to find optimal subsets of correlated coefficients for which we estimate covariances. We propose a new estimator, called MISC, that uses a mixed-integer optimization (MIO) program to find an optimal block diagonal structure specification for the covariance matrix, corresponding to subsets of correlated coefficients, for any desired sparsity level using Markov Chain Monte Carlo (MCMC) posterior draws from the unrestricted full covariance matrix. The optimal sparsity level of the covariance matrix is determined using out-of-sample validation. We demonstrate the ability of MISC to correctly recover the true covariance structure from synthetic data. In an empirical illustration using a stated preference survey on modes of transportation, we use MISC to obtain a sparse covariance matrix indicating how preferences for attributes are related to one another.

Constrained Backtracking Matching Pursuit Algorithm for Image Reconstruction in Compressed Sensing

Image reconstruction based on sparse constraints is an important research topic in compressed sensing. Sparsity adaptive matching pursuit (SAMP) is a greedy pursuit reconstruction algorithm, which reconstructs signals without prior information of the sparsity level and potentially presents better reconstruction performance than other greedy pursuit algorithms. However, SAMP still suffers from being sensitive to the step size selection at high sub-sampling ratios. To solve this problem, this paper proposes a constrained backtracking matching pursuit (CBMP) algorithm for image reconstruction. The composite strategy, including two kinds of constraints, effectively controls the increment of the estimated sparsity level at different stages and accurately estimates the true support set of images. Based on the relationship analysis between the signal and measurement, an energy criterion is also proposed as a constraint. At the same time, the four-to-one rule is improved as an extra constraint. Comprehensive experimental results demonstrate that the proposed CBMP yields better performance and further stability than other greedy pursuit algorithms for image reconstruction.

Differentiable Subset Pruning of Transformer Heads

Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model; such pruning leads to models that are noticeably smaller and faster in practice. Our work introduces a new head pruning technique that we term differentiable subset pruning. ntuitively, our method learns per- head importance variables and then enforces a user-specified hard constraint on the number of unpruned heads. he importance variables are learned via stochastic gradient descent. e conduct experiments on natural language inference and machine translation; we show that differentiable subset pruning performs comparably or better than previous works while offering precise control of the sparsity level.1

A polynomial algorithm for best-subset selection problem

Best-subset selection aims to find a small subset of predictors, so that the resulting linear model is expected to have the most desirable prediction accuracy. It is not only important and imperative in regression analysis but also has far-reaching applications in every facet of research, including computer science and medicine. We introduce a polynomial algorithm, which, under mild conditions, solves the problem. This algorithm exploits the idea of sequencing and splicing to reach a stable solution in finite steps when the sparsity level of the model is fixed but unknown. We define an information criterion that helps the algorithm select the true sparsity level with a high probability. We show that when the algorithm produces a stable optimal solution, that solution is the oracle estimator of the true parameters with probability one. We also demonstrate the power of the algorithm in several numerical studies.