Differentially private optimization algorithms for deep neural networks

2020 ◽

Vol 34 (13) ◽

pp. 2052013 ◽

Cited By ~ 3

Author(s):

Derya Soydaner

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Deep Neural Networks ◽

Optimization Algorithms ◽

Gradient Methods ◽

The Past ◽

Basic Optimization ◽

In The Wild ◽

Image Datasets

In recent years, we have witnessed the rise of deep learning. Deep neural networks have proved their success in many areas. However, the optimization of these networks has become more difficult as neural networks going deeper and datasets becoming bigger. Therefore, more advanced optimization algorithms have been proposed over the past years. In this study, widely used optimization algorithms for deep learning are examined in detail. To this end, these algorithms called adaptive gradient methods are implemented for both supervised and unsupervised tasks. The behavior of the algorithms during training and results on four image datasets, namely, MNIST, CIFAR-10, Kaggle Flowers and Labeled Faces in the Wild are compared by pointing out their differences against basic optimization algorithms.

Download Full-text

A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited–memory BFGS optimization algorithms

Neurocomputing ◽

10.1016/j.neucom.2017.05.061 ◽

2017 ◽

Vol 266 ◽

pp. 506-526 ◽

Cited By ~ 43

Author(s):

Hasan Badem ◽

Alper Basturk ◽

Abdullah Caliskan ◽

Mehmet Emin Yuksel

Keyword(s):

Neural Networks ◽

Artificial Bee Colony ◽

Deep Neural Networks ◽

Optimization Algorithms ◽

Training Strategy ◽

Limited Memory ◽

Bee Colony

Download Full-text

Training Deep Neural Networks Using Conjugate Gradient-like Methods

Electronics ◽

10.3390/electronics9111809 ◽

2020 ◽

Vol 9 (11) ◽

pp. 1809

Author(s):

Hideaki Iiduka ◽

Yu Kobayashi

Keyword(s):

Neural Networks ◽

Nonconvex Optimization ◽

Adaptive Learning ◽

Stationary Point ◽

Optimization Problem ◽

Deep Neural Networks ◽

Optimization Algorithms ◽

Learning Rate ◽

Adaptive Learning Rate ◽

Rate Optimization

The goal of this article is to train deep neural networks that accelerate useful adaptive learning rate optimization algorithms such as AdaGrad, RMSProp, Adam, and AMSGrad. To reach this goal, we devise an iterative algorithm combining the existing adaptive learning rate optimization algorithms with conjugate gradient-like methods, which are useful for constrained optimization. Convergence analyses show that the proposed algorithm with a small constant learning rate approximates a stationary point of a nonconvex optimization problem in deep learning. Furthermore, it is shown that the proposed algorithm with diminishing learning rates converges to a stationary point of the nonconvex optimization problem. The convergence and performance of the algorithm are demonstrated through numerical comparisons with the existing adaptive learning rate optimization algorithms for image and text classification. The numerical results show that the proposed algorithm with a constant learning rate is superior for training neural networks.

Download Full-text

DeepQGHO: Quantized Greedy Hyperparameter Optimization in Deep Neural Networks for on-the-fly Learning

10.21203/rs.3.rs-1146054/v1 ◽

2021 ◽

Author(s):

Anjir Ahmed Chowdhury ◽

Md Abir Hossen ◽

Md Ali Azam ◽

Md. Hafizur Rahman

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Energy Consumption ◽

Deep Neural Networks ◽

State Of The Art ◽

Optimization Algorithms ◽

Computation Time ◽

Hyperparameter Optimization ◽

Computationally Expensive ◽

Time And Energy

Abstract Hyperparameter optimization or tuning plays a significant role in the performance and reliability of deep learning (DL). Many hyperparameter optimization algorithms have been developed for obtaining better validation accuracy in DL training. Most state-of-the-art hyperparameters are computationally expensive due to a focus on validation accuracy. Therefore, they are unsuitable for online or on-the-fly training applications which require computational efficiency. In this paper, we develop a novel greedy approach-based hyperparameter optimization (GHO) algorithm for faster training applications, e.g., on-the-fly training. We perform an empirical study to compute the performance such as computation time and energy consumption of the GHO and compare it with two state-of-the-art hyperparameter optimization algorithms. We also deploy the GHO algorithm in an edge device to validate the performance of our algorithm. We perform post-training quantization to the GHO algorithm to reduce inference time and latency.

Download Full-text

Inverse Design for Silicon Photonics: From Iterative Optimization Algorithms to Deep Neural Networks

Applied Sciences ◽

10.3390/app11093822 ◽

2021 ◽

Vol 11 (9) ◽

pp. 3822

Author(s):

Simei Mao ◽

Lirong Cheng ◽

Caiyue Zhao ◽

Faisal Nadeem Khan ◽

Qian Li ◽

...

Keyword(s):

Neural Networks ◽

Silicon Photonics ◽

Design Methodology ◽

Degrees Of Freedom ◽

Deep Neural Networks ◽

Low Cost ◽

Optimization Algorithms ◽

Optimization Methods ◽

Inverse Design ◽

Iterative Optimization

Silicon photonics is a low-cost and versatile platform for various applications. For design of silicon photonic devices, the light-material interaction within its complex subwavelength geometry is difficult to investigate analytically and therefore numerical simulations are majorly adopted. To make the design process more time-efficient and to improve the device performance to its physical limits, various methods have been proposed over the past few years to manipulate the geometries of silicon platform for specific applications. In this review paper, we summarize the design methodologies for silicon photonics including iterative optimization algorithms and deep neural networks. In case of iterative optimization methods, we discuss them in different scenarios in the sequence of increased degrees of freedom: empirical structure, QR-code like structure and irregular structure. We also review inverse design approaches assisted by deep neural networks, which generate multiple devices with similar structure much faster than iterative optimization methods and are thus suitable in situations where piles of optical components are needed. Finally, the applications of inverse design methodology in optical neural networks are also discussed. This review intends to provide the readers with the suggestion for the most suitable design methodology for a specific scenario.

Download Full-text

Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks

IEEE Transactions on Cybernetics ◽

10.1109/tcyb.2021.3107415 ◽

2021 ◽

pp. 1-12

Author(s):

Hideaki Iiduka

Keyword(s):

Neural Networks ◽

Adaptive Learning ◽

Deep Neural Networks ◽

Optimization Algorithms ◽

Learning Rate ◽

Adaptive Learning Rate ◽

Learning Rates ◽

Rate Optimization

Download Full-text

Automated Design of Deep Neural Networks

ACM Computing Surveys ◽

10.1145/3439730 ◽

2021 ◽

Vol 54 (2) ◽

pp. 1-37

Author(s):

El-Ghazali Talbi

Keyword(s):

Neural Networks ◽

Large Scale ◽

Deep Neural Networks ◽

Optimization Algorithms ◽

Search Space ◽

Parallel Optimization ◽

Objective Functions ◽

Mixed Design ◽

Comprehensive Survey ◽

Variation Operators

In recent years, research in applying optimization approaches in the automatic design of deep neural networks has become increasingly popular. Although various approaches have been proposed, there is a lack of a comprehensive survey and taxonomy on this hot research topic. In this article, we propose a unified way to describe the various optimization algorithms that focus on common and important search components of optimization algorithms: representation, objective function, constraints, initial solution(s), and variation operators. In addition to large-scale search space, the problem is characterized by its variable mixed design space, it is very expensive, and it has multiple blackbox objective functions. Hence, this unified methodology has been extended to advanced optimization approaches, such as surrogate-based, multi-objective, and parallel optimization.

Download Full-text