scholarly journals Proximal average approximated incremental gradient descent for composite penalty regularized empirical risk minimization

2016 ◽  
Vol 106 (4) ◽  
pp. 595-622 ◽  
Author(s):  
Yiu-ming Cheung ◽  
Jian Lou
Keyword(s):  
Gradient Descent ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Proximal Average ◽  
Regularized Empirical Risk Minimization ◽  
Incremental Gradient Descent

scholarly journals Efficient Private ERM for Smooth Objectives

2017 ◽  
Author(s):  
Jiaqi Zhang ◽  
Kai Zheng ◽  
Wenlong Mou ◽  
Liwei Wang
Keyword(s):  
Gradient Descent ◽  
Optimization Algorithms ◽  
Stochastic Gradient Descent ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Running Time ◽  
Strongly Convex ◽  
Empirical Risk ◽  
Gradient Descent Algorithm ◽  
Previous State

In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running time of previous state-of-the-art private optimization algorithms, for both $\epsilon$-DP and $(\epsilon, \delta)$-DP. For non-convex but smooth objectives, we propose an RRPSGD (Random Round Private Stochastic Gradient Descent) algorithm, which provably converges to a stationary point with privacy guarantee. Besides the expected utility bounds, we also provide guarantees in high probability form. Experiments demonstrate that our algorithm consistently outperforms existing method in both utility and running time.

scholarly journals On Concentration for (Regularized) Empirical Risk Minimization

Sankhya A ◽  
2017 ◽  
Vol 79 (2) ◽  
pp. 159-200 ◽  
Author(s):  
Sara van de Geer ◽  
Martin J. Wainwright
Keyword(s):  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Regularized Empirical Risk Minimization

scholarly journals Distributed Accelerated Proximal Coordinate Gradient Methods

2017 ◽  
Author(s):  
Yong Ren ◽  
Jun Zhu
Keyword(s):  
Optimization Problem ◽  
Gradient Methods ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
L1 Norm ◽  
Coordinate Descent Algorithm ◽  
Empirical Risk ◽  
Parallel Structures ◽  
Regularized Empirical Risk Minimization ◽  
Special Case

We develop a general accelerated proximal coordinate descent algorithm in distributed settings (Dis- APCG) for the optimization problem that minimizes the sum of two convex functions: the first part f is smooth with a gradient oracle, and the other one Ψ is separable with respect to blocks of coordinate and has a simple known structure (e.g., L1 norm). Our algorithm gets new accelerated convergence rate in the case that f is strongly con- vex by making use of modern parallel structures, and includes previous non-strongly case as a special case. We further present efficient implementations to avoid full-dimensional operations in each step, significantly reducing the computation cost. Experiments on the regularized empirical risk minimization problem demonstrate the effectiveness of our algorithm and match our theoretical findings.

scholarly journals Adaptive Proximal Average Based Variance Reducing Stochastic Methods for Optimization with Composite Regularization

2019 ◽  
Vol 33 ◽  
pp. 1552-1559
Author(s):  
Jingchang Liu ◽  
Linli Xu ◽  
Junliang Guo ◽  
Xin Sheng
Keyword(s):  
Structural Information ◽  
Stochastic Methods ◽  
Empirical Risk Minimization ◽  
Initial Number ◽  
Risk Minimization ◽  
Proximal Operator ◽  
Empirical Risk ◽  
Proximal Average ◽  
Learning Tasks ◽  
Operator Calculation

We focus on empirical risk minimization with a composite regulariser, which has been widely applied in various machine learning tasks to introduce important structural information regarding the problem or data. In general, it is challenging to calculate the proximal operator with the composite regulariser. Recently, proximal average (PA) which involves a feasible proximal operator calculation is proposed to approximate composite regularisers. Augmented with the prevailing variance reducing (VR) stochastic methods (e.g. SVRG, SAGA), PA based algorithms would achieve a better performance. However, existing works require a fixed stepsize, which needs to be rather small to ensure that the PA approximation is sufficiently accurate. In the meantime, the smaller stepsize would incur many more iterations for convergence. In this paper, we propose two fast PA based VR stochastic methods – APA-SVRG and APA-SAGA. By initializing the stepsize with a much larger value and adaptively decreasing it, both of the proposed methods are proved to enjoy the (ô n log 1/ε + mo 1/ε) iteration complexity to achieve the accurate solutions, where m0 is the initial number of inner iterations and n is the number of samples. Moreover, experimental results demonstrate the superiority of the proposed algorithms.

Sign in / Sign up

Export Citation Format

Share Document