scholarly journals Efficient Private ERM for Smooth Objectives

2017 ◽  
Author(s):  
Jiaqi Zhang ◽  
Kai Zheng ◽  
Wenlong Mou ◽  
Liwei Wang
Keyword(s):  
Gradient Descent ◽  
Optimization Algorithms ◽  
Stochastic Gradient Descent ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Running Time ◽  
Strongly Convex ◽  
Empirical Risk ◽  
Gradient Descent Algorithm ◽  
Previous State

In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running time of previous state-of-the-art private optimization algorithms, for both $\epsilon$-DP and $(\epsilon, \delta)$-DP. For non-convex but smooth objectives, we propose an RRPSGD (Random Round Private Stochastic Gradient Descent) algorithm, which provably converges to a stationary point with privacy guarantee. Besides the expected utility bounds, we also provide guarantees in high probability form. Experiments demonstrate that our algorithm consistently outperforms existing method in both utility and running time.

scholarly journals Do Subsampled Newton Methods Work for High-Dimensional Data?

2020 ◽  
Vol 34 (04) ◽  
pp. 4723-4730
Author(s):  
Xiang Li ◽  
Shusen Wang ◽  
Zhihua Zhang
Keyword(s):  
High Dimensional Data ◽  
High Dimensional ◽  
Empirical Risk Minimization ◽  
Newton Methods ◽  
Risk Minimization ◽  
Strongly Convex ◽  
Empirical Risk ◽  
Data Points ◽  
Approximate Hessian

Subsampled Newton methods approximate Hessian matrices through subsampling techniques to alleviate the per-iteration cost. Previous results require Ω (d) samples to approximate Hessians, where d is the dimension of data points, making it less practical for high-dimensional data. The situation is deteriorated when d is comparably as large as the number of data points n, which requires to take the whole dataset into account, making subsampling not useful. This paper theoretically justifies the effectiveness of subsampled Newton methods on strongly convex empirical risk minimization with high dimensional data. Specifically, we provably require only Θ˜(deffγ) samples for approximating the Hessian matrices, where deffγ is the γ-ridge leverage and can be much smaller than d as long as nγ ≫ 1. Our theories work for three types of Newton methods: subsampled Netwon, distributed Newton, and proximal Newton.

scholarly journals Differentially Private Learning with Small Public Data

2020 ◽  
Vol 34 (04) ◽  
pp. 6219-6226
Author(s):  
Jun Wang ◽  
Zhi-Hua Zhou
Keyword(s):  
Gradient Descent ◽  
Differential Privacy ◽  
Public Information ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Private Data ◽  
Public Data ◽  
Common Situation

Differentially private learning tackles tasks where the data are private and the learning process is subject to differential privacy requirements. In real applications, however, some public data are generally available in addition to private data, and it is interesting to consider how to exploit them. In this paper, we study a common situation where a small amount of public data can be used when solving the Empirical Risk Minimization problem over a private database. Specifically, we propose Private-Public Stochastic Gradient Descent, which utilizes such public information to adjust parameters in differentially private stochastic gradient descent and fine-tunes the final result with model reuse. Our method keeps differential privacy for the private database, and empirical study validates its superiority compared with existing approaches.

scholarly journals Distributed block-diagonal approximation methods for regularized empirical risk minimization

2019 ◽  
Vol 109 (4) ◽  
pp. 813-852
Author(s):  
Ching-pei Lee ◽  
Kai-Wei Chang
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Broad Class ◽  
Linear Convergence ◽  
Approximate Solutions ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Global Linear Convergence ◽  
Strongly Convex ◽  
Empirical Risk

AbstractIn recent years, there is a growing need to train machine learning models on a huge volume of data. Therefore, designing efficient distributed optimization algorithms for empirical risk minimization (ERM) has become an active and challenging research topic. In this paper, we propose a flexible framework for distributed ERM training through solving the dual problem, which provides a unified description and comparison of existing methods. Our approach requires only approximate solutions of the sub-problems involved in the optimization process, and is versatile to be applied on many large-scale machine learning problems including classification, regression, and structured prediction. We show that our framework enjoys global linear convergence for a broad class of non-strongly-convex problems, and some specific choices of the sub-problems can even achieve much faster convergence than existing approaches by a refined analysis. This improved convergence rate is also reflected in the superior empirical performance of our method.

scholarly journals Revisiting the dynamic interactions between economic growth and environmental pollution in Italy: evidence from a gradient descent algorithm

2021 ◽  
Author(s):  
Marco Mele ◽  
Cosimo Magazzino ◽  
Nicolas Schneider ◽  
Floriana Nicolai
Keyword(s):  
Economic Growth ◽  
Gradient Descent ◽  
Statistical Data ◽  
Stochastic Gradient Descent ◽  
Policy Recommendations ◽  
Dynamic Interactions ◽  
Gradient Descent Algorithm ◽  
The Relationship ◽  
Main Strand ◽  
Yearly Data

AbstractAlthough the literature on the relationship between economic growth and CO2 emissions is extensive, the use of machine learning (ML) tools remains seminal. In this paper, we assess this nexus for Italy using innovative algorithms, with yearly data for the 1960–2017 period. We develop three distinct models: the batch gradient descent (BGD), the stochastic gradient descent (SGD), and the multilayer perceptron (MLP). Despite the phase of low Italian economic growth, results reveal that CO2 emissions increased in the predicting model. Compared to the observed statistical data, the algorithm shows a correlation between low growth and higher CO2 increase, which contradicts the main strand of literature. Based on this outcome, adequate policy recommendations are provided.

scholarly journals Learning Bounds of ERM Principle for Sequences of Time-Dependent Samples

2015 ◽  
Vol 2015 ◽  
pp. 1-8
Author(s):  
Mingchen Yao ◽  
Chao Zhang ◽  
Wei Wu
Keyword(s):  
Time Dependent ◽  
Dependent Data ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Practical Applications ◽  
Empirical Risk ◽  
Learning Tasks ◽  
Generalization Bound ◽  
Erm Principle ◽  
Dependent Samples

Many generalization results in learning theory are established under the assumption that samples are independent and identically distributed (i.i.d.). However, numerous learning tasks in practical applications involve the time-dependent data. In this paper, we propose a theoretical framework to analyze the generalization performance of the empirical risk minimization (ERM) principle for sequences of time-dependent samples (TDS). In particular, we first present the generalization bound of ERM principle for TDS. By introducing some auxiliary quantities, we also give a further analysis of the generalization properties and the asymptotical behaviors of ERM principle for TDS.

Differentially Private Empirical Risk Minimization for AUC Maximization

2021 ◽  
Author(s):  
Puyu Wang ◽  
Zhenhuan Yang ◽  
Yunwen Lei ◽  
Yiming Ying ◽  
Hai Zhang
Keyword(s):  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Auc Maximization

Soft-Sign Stochastic Gradient Descent Algorithm for Wireless Federated Learning

2021 ◽  
Author(s):  
Seunghoon Lee ◽  
Chanho Park ◽  
Songnam Hong ◽  
Yonina C. Eldar ◽  
Namyoon Lee
Keyword(s):  
Gradient Descent ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Descent Algorithm ◽  
Gradient Descent Algorithm
Sign in / Sign up

Export Citation Format

Share Document