scholarly journals Differentially Private Learning with Small Public Data

2020 ◽  
Vol 34 (04) ◽  
pp. 6219-6226
Author(s):  
Jun Wang ◽  
Zhi-Hua Zhou
Keyword(s):  
Gradient Descent ◽  
Differential Privacy ◽  
Public Information ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Private Data ◽  
Public Data ◽  
Common Situation

Differentially private learning tackles tasks where the data are private and the learning process is subject to differential privacy requirements. In real applications, however, some public data are generally available in addition to private data, and it is interesting to consider how to exploit them. In this paper, we study a common situation where a small amount of public data can be used when solving the Empirical Risk Minimization problem over a private database. Specifically, we propose Private-Public Stochastic Gradient Descent, which utilizes such public information to adjust parameters in differentially private stochastic gradient descent and fine-tunes the final result with model reuse. Our method keeps differential privacy for the private database, and empirical study validates its superiority compared with existing approaches.

scholarly journals Efficient Private ERM for Smooth Objectives

2017 ◽  
Author(s):  
Jiaqi Zhang ◽  
Kai Zheng ◽  
Wenlong Mou ◽  
Liwei Wang
Keyword(s):  
Gradient Descent ◽  
Optimization Algorithms ◽  
Stochastic Gradient Descent ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Running Time ◽  
Strongly Convex ◽  
Empirical Risk ◽  
Gradient Descent Algorithm ◽  
Previous State

In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running time of previous state-of-the-art private optimization algorithms, for both $\epsilon$-DP and $(\epsilon, \delta)$-DP. For non-convex but smooth objectives, we propose an RRPSGD (Random Round Private Stochastic Gradient Descent) algorithm, which provably converges to a stationary point with privacy guarantee. Besides the expected utility bounds, we also provide guarantees in high probability form. Experiments demonstrate that our algorithm consistently outperforms existing method in both utility and running time.

scholarly journals Entropic gradient descent algorithms and wide flat minima*

2021 ◽  
Vol 2021 (12) ◽  
pp. 124015
Author(s):  
Fabrizio Pittorino ◽  
Carlo Lucibello ◽  
Christoph Feinauer ◽  
Gabriele Perugini ◽  
Carlo Baldassi ◽  
...  
Keyword(s):  
Gradient Descent ◽  
State Of The Art ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Generalization Error ◽  
Local Entropy ◽  
Empirical Risk ◽  
Alternative Measures ◽  
Two Measures ◽  
The Relationship

Abstract The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. In this work we first discuss the relationship between alternative measures of flatness: the local entropy, which is useful for analysis and algorithm development, and the local energy, which is easier to compute and was shown empirically in extensive tests on state-of-the-art networks to be the best predictor of generalization capabilities. We show semi-analytically in simple controlled scenarios that these two measures correlate strongly with each other and with generalization. Then, we extend the analysis to the deep learning scenario by extensive numerical validations. We study two algorithms, entropy-stochastic gradient descent and replicated-stochastic gradient descent, that explicitly include the local entropy in the optimization objective. We devise a training schedule by which we consistently find flatter minima (using both flatness measures), and improve the generalization error for common architectures (e.g. ResNet, EfficientNet).

Sign in / Sign up

Export Citation Format

Share Document