scholarly journals Adaptive Stochastic Variance Reduction for Subsampled Newton Method with Cubic Regularization

Author(s):  
Junyu Zhang ◽  
Lin Xiao ◽  
Shuzhong Zhang

The cubic regularized Newton method of Nesterov and Polyak has become increasingly popular for nonconvex optimization because of its capability of finding an approximate local solution with a second order guarantee and its low iteration complexity. Several recent works extend this method to the setting of minimizing the average of N smooth functions by replacing the exact gradients and Hessians with subsampled approximations. It is shown that the total Hessian sample complexity can be reduced to be sublinear in N per iteration by leveraging stochastic variance reduction techniques. We present an adaptive variance reduction scheme for a subsampled Newton method with cubic regularization and show that the expected Hessian sample complexity is [Formula: see text] for finding an [Formula: see text]-approximate local solution (in terms of first and second order guarantees, respectively). Moreover, we show that the same Hessian sample complexity is retained with fixed sample sizes if exact gradients are used. The techniques of our analysis are different from previous works in that we do not rely on high probability bounds based on matrix concentration inequalities. Instead, we derive and utilize new bounds on the third and fourth order moments of the average of random matrices, which are of independent interest on their own.

1995 ◽  
Vol 25 (11) ◽  
pp. 1783-1794 ◽  
Author(s):  
Thomas B. Lynch

Three basic techniques are proposed for reducing the variance of the stand volume estimate provided by cylinder sampling and Ueno's method. Ueno's method is based on critical height sampling but does not require measurement of critical heights. Instead, a count of trees whose critical heights are less than randomly generated heights is used to estimate stand volume. Cylinder sampling selects sample trees for which randomly generated heights fall within cylinders formed by tree heights and point sampling plot sizes. The methods proposed here for variance reduction in cylinder sampling and Ueno's method are antithetic variates, importance sampling, and control variates. Cylinder sampling without variance reduction was the most efficient of 12 methods compared in computer simulation that used estimated measurement times. However, cylinder sampling requires knowledge of a combined variable individual tree volume equation. Of the three variance reduction techniques applied to Ueno's method, antithetic variates performed best in computer simulation.


2018 ◽  
Vol 24 (2) ◽  
pp. 101-115 ◽  
Author(s):  
Mohamed-Slim Alouini ◽  
Nadhir Ben Rached ◽  
Abla Kammoun ◽  
Raul Tempone

Abstract The sum of log-normal variates is encountered in many challenging applications such as performance analysis of wireless communication systems and financial engineering. Several approximation methods have been reported in the literature. However, these methods are not accurate in the tail regions. These regions are of primordial interest as small probability values have to be evaluated with high precision. Variance reduction techniques are known to yield accurate, yet efficient, estimates of small probability values. Most of the existing approaches have focused on estimating the right-tail of the sum of log-normal random variables (RVs). Here, we instead consider the left-tail of the sum of correlated log-normal variates with Gaussian copula, under a mild assumption on the covariance matrix. We propose an estimator combining an existing mean-shifting importance sampling approach with a control variate technique. This estimator has an asymptotically vanishing relative error, which represents a major finding in the context of the left-tail simulation of the sum of log-normal RVs. Finally, we perform simulations to evaluate the performances of the proposed estimator in comparison with existing ones.


Sign in / Sign up

Export Citation Format

Share Document