scholarly journals An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Xue Yu ◽  
Yifan Sun ◽  
Hai-Jun Zhou
Keyword(s):  
Linear Regression ◽  
Message Passing ◽  
Linear Models ◽  
Greedy Algorithms ◽  
Adaptive Lasso ◽  
Regression Coefficients ◽  
High Dimensional ◽  
Linear Regression Models ◽  
Approximate Message Passing ◽  
Highly Correlated

AbstractHigh-dimensional linear regression model is the most popular statistical model for high-dimensional data, but it is quite a challenging task to achieve a sparse set of regression coefficients. In this paper, we propose a simple heuristic algorithm to construct sparse high-dimensional linear regression models, which is adapted from the shortest-solution guided decimation algorithm and is referred to as ASSD. This algorithm constructs the support of regression coefficients under the guidance of the shortest least-squares solution of the recursively decimated linear models, and it applies an early-stopping criterion and a second-stage thresholding procedure to refine this support. Our extensive numerical results demonstrate that ASSD outperforms LASSO, adaptive LASSO, vector approximate message passing, and two other representative greedy algorithms in solution accuracy and robustness. ASSD is especially suitable for linear regression problems with highly correlated measurement matrices encountered in real-world applications.

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

2021 ◽  
Author(s):  
Xue Yu ◽  
Yifan Sun ◽  
Hai-Jun Zhou
Keyword(s):  
Linear Regression ◽  
Message Passing ◽  
Greedy Algorithms ◽  
Linear Equations ◽  
Regression Coefficients ◽  
High Dimensional ◽  
Linear Regression Models ◽  
Approximate Message Passing ◽  
Highly Correlated ◽  
Solution Accuracy

Abstract High-dimensional linear regression model is the most popular statistical model for high-dimensional data, but it is quite a challenging task to achieve a sparse set of regression coefficients. In this paper, we propose a simple heuristic algorithm to construct sparse high-dimensional linear regression models, which is adapted from the shortest-solution guided decimation algorithm and is referred to as ASSD. This algorithm constructs the support of regression coefficients under the guidance of the least-squares solution of the recursively decimated linear equations, and it applies an early-stopping criterion and a second-stage thresholding procedure to refine this support. Our extensive numerical results demonstrate that ASSD outper-forms LASSO, vector approximate message passing, and two other representative greedy algorithms in solution accuracy and robustness. ASSD is especially suitable for linear regression problems with highly correlated measurement matrices encountered in real-world applications.

scholarly journals Optimal errors and phase transitions in high-dimensional generalized linear models

2019 ◽  
Vol 116 (12) ◽  
pp. 5451-5460 ◽  
Author(s):  
Jean Barbier ◽  
Florent Krzakala ◽  
Nicolas Macris ◽  
Léo Miolane ◽  
Lenka Zdeborová
Keyword(s):  
Phase Transitions ◽  
Generalized Linear Models ◽  
Message Passing ◽  
Statistical Physics ◽  
Linear Models ◽  
Optimal Estimation ◽  
Error Correcting Codes ◽  
Data Matrix ◽  
High Dimensional ◽  
Special Cases

Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or “free entropy”) from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms.

Sequential Model Averaging for High Dimensional Linear Regression Models

2017 ◽  
Author(s):  
Wei Lan ◽  
Yingying Ma ◽  
Junlong Zhao ◽  
Hansheng Wang ◽  
Chih-Ling Tsai
Keyword(s):  
Linear Regression ◽  
Regression Models ◽  
Model Averaging ◽  
High Dimensional ◽  
Linear Regression Models ◽  
Sequential Model

scholarly journals Seagull: lasso, group lasso and sparse-group lasso regularization for linear regression models via proximal gradient descent

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Jan Klosa ◽  
Noah Simon ◽  
Pål Olof Westermark ◽  
Volkmar Liebscher ◽  
Dörte Wittenburg
Keyword(s):  
Linear Regression ◽  
Regression Models ◽  
Gradient Descent ◽  
Methylation Status ◽  
R Package ◽  
Group Lasso ◽  
High Dimensional ◽  
Linear Regression Models ◽  
Sparse Group Lasso ◽  
Proximal Gradient Descent

Abstract Background Statistical analyses of biological problems in life sciences often lead to high-dimensional linear models. To solve the corresponding system of equations, penalization approaches are often the methods of choice. They are especially useful in case of multicollinearity, which appears if the number of explanatory variables exceeds the number of observations or for some biological reason. Then, the model goodness of fit is penalized by some suitable function of interest. Prominent examples are the lasso, group lasso and sparse-group lasso. Here, we offer a fast and numerically cheap implementation of these operators via proximal gradient descent. The grid search for the penalty parameter is realized by warm starts. The step size between consecutive iterations is determined with backtracking line search. Finally, seagull -the R package presented here- produces complete regularization paths. Results Publicly available high-dimensional methylation data are used to compare seagull to the established R package SGL. The results of both packages enabled a precise prediction of biological age from DNA methylation status. But even though the results of seagull and SGL were very similar (R2 > 0.99), seagull computed the solution in a fraction of the time needed by SGL. Additionally, seagull enables the incorporation of weights for each penalized feature. Conclusions The following operators for linear regression models are available in seagull: lasso, group lasso, sparse-group lasso and Integrative LASSO with Penalty Factors (IPF-lasso). Thus, seagull is a convenient envelope of lasso variants.

Sign in / Sign up

Export Citation Format

Share Document