STRETCHING THE NET: MULTIDIMENSIONAL REGULARIZATION

2021 ◽  
pp. 1-30
Author(s):  
Jaume Vives-i-Bastida
Keyword(s):  
Empirical Bayes ◽  
Cross Validation ◽  
Elastic Net ◽  
Shrinkage Estimators ◽  
Expected Loss ◽  
Bayes Risk ◽  
Asymptotic Risk ◽  
Regularization Parameters ◽  
Unbiased Risk ◽  
Sparsity Level

This paper derives asymptotic risk (expected loss) results for shrinkage estimators with multidimensional regularization in high-dimensional settings. We introduce a class of multidimensional shrinkage estimators (MuSEs), which includes the elastic net, and show that—as the number of parameters to estimate grows—the empirical loss converges to the oracle-optimal risk. This result holds when the regularization parameters are estimated empirically via cross-validation or Stein’s unbiased risk estimate. To help guide applied researchers in their choice of estimator, we compare the empirical Bayes risk of the lasso, ridge, and elastic net in a spike and normal setting. Of the three estimators, we find that the elastic net performs best when the data are moderately sparse and the lasso performs best when the data are highly sparse. Our analysis suggests that applied researchers who are unsure about the level of sparsity in their data might benefit from using MuSEs such as the elastic net. We exploit these insights to propose a new estimator, the cubic net, and demonstrate through simulations that it outperforms the three other estimators for any sparsity level.

scholarly journals Dependence-Maximization Clustering with Least-Squares Mutual Information

2011 ◽  
Vol 15 (7) ◽  
pp. 800-805 ◽  
Author(s):  
Manabu Kimura ◽  
◽  
Masashi Sugiyama
Keyword(s):  
Mutual Information ◽  
Least Squares ◽  
Cross Validation ◽  
Unsupervised Clustering ◽  
Statistical Dependence ◽  
Clustering Method ◽  
Regularization Parameters ◽  
Loss Variant ◽  
Dependence Measures ◽  
Kernel Parameters

Recently, statistical dependence measures such as mutual information and kernelized covariance have been successfully applied to clustering. In this paper, we follow this line of research and propose a novel dependence-maximization clustering method based on least-squares mutual information, which is an estimator of a squared-loss variant of mutual information. A notable advantage of the proposed method over existing approaches is that hyperparameters such as kernel parameters and regularization parameters can be objectively optimized based on cross-validation. Thus, subjective manual-tuning of hyperparameters is not necessary in the proposed method, which is a highly useful property in unsupervised clustering scenarios. Through experiments, we illustrate the usefulness of the proposed approach.

Optimal Regularization Methods for Inverse Heat Transfer Problems

Volume 4 ◽  
2004 ◽  
Author(s):  
Kei Okamoto ◽  
Ben Q. Li
Keyword(s):  
Heat Transfer ◽  
Heat Flux ◽  
Regularization Method ◽  
Cross Validation ◽  
Tikhonov Regularization Method ◽  
Inverse Algorithm ◽  
Marquardt Method ◽  
Inverse Heat Transfer ◽  
Regularization Parameters ◽  
Levenberg Marquardt

The Tikhonov regularization method has been used to find the unknown heat flux distribution along the boundary when the temperature measurements are known in the interior of a sample. Mathematically, the inverse problem is ill-posed, though physically correct, and prone to instability. This paper discusses the fundamental issues concerning the selection of optimal regularization parameters for inverse heat transfer calculations. Towards this end, a finite-element-based inverse algorithm is developed. Five different methods, that is, the maximum likelihood (ML), the ordinary cross-validation (OCV), the generalized cross-validation (GCV), the L-curve method, and the discrepancy principle, are evaluated for the purpose of determining optimal regularization parameters. An assessment of these methods is made using 1-D and 2-D inverse steady heat conduction problems where analytical solutions are available. The optimal regularization method is also compared with the Levenberg-Marquardt method for inverse heat transfer calculations. Results show that in general the Tikhonov regularization method is superior over the Levenberg-Marquardt method when the input data errors are noisy. With the appropriately determined regularization parameter, the inverse algorithm is applied to estimate the heat flux of spray cooling of a 3-D microelectronic component with an embedded heating source.

scholarly journals Assessing regularised solutions

1988 ◽  
Vol 30 (1) ◽  
pp. 24-42 ◽  
Author(s):  
M. A. Lukas
Keyword(s):  
Cross Validation ◽  
Noisy Data ◽  
Optimal Filtering ◽  
Optimal Convergence Rate ◽  
Error Norm ◽  
Optimal Sequence ◽  
True Solution ◽  
Optimal Convergence ◽  
Ill Posed ◽  
Unbiased Risk

AbstractConsider the prototype ill-posed problem of a first kind integral equation ℛ with discrete noisy data di, = f(xi) + εi, i = 1, …, n. Let u0 be the true solution and unα a regularised solution with regularisation parameter α. Under certain assumptions, it is known that if α → 0 but not too quickly as n → ∞, then unα converges to u0. We examine the dependence of the optimal sequence of α and resulting optimal convergence rate on the smoothness of f or u0, the kernel K, the order of regularisation m and the error norm used. Some important implications are made, including the fact that m must be sufficiently high relative to the smoothness of u0 in order to ensure optimal convergence. An optimal filtering criterion is used to determine the order where is the maximum smoothness of u0. Two practical methods for estimating the optimal α, the unbiased risk estimate and generalised cross validation, are also discussed.

scholarly journals An Empirical Bayes Approach to Estimating Dynamic Models of Co-regulated Gene Expression

2021 ◽  
Author(s):  
Sara Venkatraman ◽  
Sumanta Basu ◽  
Andrew G. Clark ◽  
Sofie Y.N. Delbare ◽  
Myung Hee Lee ◽  
...  
Keyword(s):  
Gene Expression ◽  
Gene Networks ◽  
Empirical Bayes ◽  
Dynamic Models ◽  
Expression Patterns ◽  
Similarity Metrics ◽  
Empirical Bayes Approach ◽  
Unbiased Risk ◽  
Regulated Gene Expression ◽  
Bayes Approach

Time-course gene expression datasets provide insight into the dynamics of complex biological processes, such as immune response and organ development. It is of interest to identify genes with similar temporal expression patterns because such genes are often biologically related. However, this task is challenging due to the high dimensionality of such datasets and the nonlinearity of gene expression time dynamics. We propose an empirical Bayes approach to estimating ordinary differential equation (ODE) models of gene expression, from which we derive similarity metrics that can be used to identify groups of genes with co-moving or time-delayed expression patterns. These metrics, which we call the Bayesian lead-lag R2 values, can be used to construct clusters or networks of functionally-related genes. A key feature of this method is that it leverages biological databases that document known interactions between genes. This information is automatically used to define informative prior distributions on the ODE model's parameters. We then derive data-driven shrinkage parameters from Stein's unbiased risk estimate that optimally balance the ODE model's fit to both the data and external biological information. Using real gene expression data, we demonstrate that our biologically-informed similarity metrics allow us to recover sparse, interpretable gene networks. These networks reveal new insights about the dynamics of biological systems.

scholarly journals Methods of plant breeding in the genome era

2010 ◽  
Vol 92 (5-6) ◽  
pp. 423-441 ◽  
Author(s):  
SHIZHONG XU ◽  
ZHIQIU HU
Keyword(s):  
Plant Breeding ◽  
Least Squares ◽  
Empirical Bayes ◽  
Mixed Model ◽  
Cross Validation ◽  
Probability Distributions ◽  
Data Set ◽  
Bayesian Shrinkage ◽  
Accuracy Of Prediction ◽  
Marker Information

SummaryMethods of genomic value prediction are reviewed. The majority of the methods are related to mixed model methodology, either explicitly or implicitly, by treating systematic environmental effects as fixed and quantitative trait locus (QTL) effects as random. Six different methods are reviewed, including least squares (LS), ridge regression, Bayesian shrinkage, least absolute shrinkage and selection operator (Lasso), empirical Bayes and partial least squares (PLS). The LS and PLS methods are non-Bayesian because they do not require probability distributions for the data. The PLS method is introduced as a special dimension reduction scheme to handle high-density marker information. Theory and methods of cross-validation are described. The leave-one-out cross-validation approach is recommended for model validation. A working example is used to demonstrate the utility of genome selection (GS) in barley. The data set contained 150 double haploid lines and 495 DNA markers covering the entire barley genome, with an average marker interval of 2·23 cM. Eight quantitative traits were included in the analysis. GS using the empirical Bayesian method showed high predictability of the markers for all eight traits with a mean accuracy of prediction of 0·70. With traditional marker-assisted selection (MAS), the average accuracy of prediction was 0·59, giving an average gain of GS over MAS of 0·11. This study provided strong evidence that GS using marker information alone can be an efficient tool for plant breeding.

Bayes and Empirical Bayes Shrinkage Estimation of Regression Coefficients: A Cross-Validation Study

1988 ◽  
Vol 13 (3) ◽  
pp. 199-213 ◽  
Author(s):  
Fassii Nebebe ◽  
T. W. F. Stroud
Keyword(s):  
Least Squares ◽  
Empirical Bayes ◽  
Cross Validation ◽  
Regression Coefficients ◽  
Data Sets ◽  
Shrinkage Estimation ◽  
Bayes Methods ◽  
Least Absolute Deviations ◽  
Empirical Bayes Methods ◽  
Empirical Bayes Approach

A Bayesian and an empirical Bayes approach to shrinkage estimation of regression coefficients and the uses of these in prediction are investigated. The methods, along with least squares and least absolute deviations, are applied to data sets of different sizes and cross-validated with observations not in the data sets. The fully Bayes and empirical Bayes methods are seen to perform consistently better in predicting the response variable than either of least squares or least absolute deviations.

Sign in / Sign up

Export Citation Format

Share Document