A REVIEW OF OPTIMIZATION METHODS IN DEEP LEARNING

2018 ◽  
Vol 6 (4) ◽  
pp. 440-447
Author(s):  
Amita Khatana ◽  
◽  
◽  
◽  
V.K Narang ◽  
...  
Keyword(s):  
Deep Learning ◽  
Optimization Methods

scholarly journals Thermal-based early breast cancer detection using inception V3, inception V4 and modified inception MV4

2021 ◽  
Author(s):  
Mohammed Abdulla Salim Al Husaini ◽  
Mohamed Hadi Habaebi ◽  
Teddy Surya Gunawan ◽  
Md Rafiqul Islam ◽  
Elfatih A. A. Elsheikh ◽  
...  
Keyword(s):  
Breast Cancer ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Early Stage ◽  
Computational Cost ◽  
Optimization Methods ◽  
Optimization Method ◽  
Deep Convolutional Neural Networks ◽  
Breast Thermography

AbstractBreast cancer is one of the most significant causes of death for women around the world. Breast thermography supported by deep convolutional neural networks is expected to contribute significantly to early detection and facilitate treatment at an early stage. The goal of this study is to investigate the behavior of different recent deep learning methods for identifying breast disorders. To evaluate our proposal, we built classifiers based on deep convolutional neural networks modelling inception V3, inception V4, and a modified version of the latter called inception MV4. MV4 was introduced to maintain the computational cost across all layers by making the resultant number of features and the number of pixel positions equal. DMR database was used for these deep learning models in classifying thermal images of healthy and sick patients. A set of epochs 3–30 were used in conjunction with learning rates 1 × 10–3, 1 × 10–4 and 1 × 10–5, Minibatch 10 and different optimization methods. The training results showed that inception V4 and MV4 with color images, a learning rate of 1 × 10–4, and SGDM optimization method, reached very high accuracy, verified through several experimental repetitions. With grayscale images, inception V3 outperforms V4 and MV4 by a considerable accuracy margin, for any optimization methods. In fact, the inception V3 (grayscale) performance is almost comparable to inception V4 and MV4 (color) performance but only after 20–30 epochs. inception MV4 achieved 7% faster classification response time compared to V4. The use of MV4 model is found to contribute to saving energy consumed and fluidity in arithmetic operations for the graphic processor. The results also indicate that increasing the number of layers may not necessarily be useful in improving the performance.

scholarly journals Neural networks for topology optimization

2019 ◽  
Vol 34 (4) ◽  
pp. 215-223 ◽  
Author(s):  
Ivan Sosnovik ◽  
Ivan Oseledets
Keyword(s):  
Deep Learning ◽  
Topology Optimization ◽  
High Performance ◽  
Current Method ◽  
Optimization Methods ◽  
Proposed Model ◽  
Significant Acceleration ◽  
Decoder Architecture ◽  
Segmentation Task ◽  
Convolutional Encoder

Abstract In this research, we propose a deep learning based approach for speeding up the topology optimization methods. The problem we seek to solve is the layout problem. The main novelty of this work is to state the problem as an image segmentation task. We leverage the power of deep learning methods as the efficient pixel-wise image labeling technique to perform the topology optimization. We introduce convolutional encoder-decoder architecture and the overall approach of solving the above-described problem with high performance. The conducted experiments demonstrate the significant acceleration of the optimization process. The proposed approach has excellent generalization properties. We demonstrate the ability of the application of the proposed model to other problems. The successful results, as well as the drawbacks of the current method, are discussed.

scholarly journals The general design of the automation for multiple fields using reinforcement learning algorithm

2022 ◽  
Vol 25 (1) ◽  
pp. 481
Author(s):  
Vijaya Kumar Reddy Radha ◽  
Anantha N. Lakshmipathi ◽  
Ravi Kumar Tirandasu ◽  
Paruchuri Ravi Prakash
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Optimization Methods ◽  
Graph Representation ◽  
Machine Learning Technique ◽  
Learning Society ◽  
Learning Technique ◽  
Learning Concept

<p>Reinforcement learning is considered as a machine learning technique that is anxious with software agents should behave in particular environment. Reinforcement learning (RL) is a division of deep learning concept that assists you to make best use of some part of the collective return. In this paper evolving reinforcement learning algorithms shows possible to learn a fresh and understable concept by using a graph representation and applying optimization methods from the auto machine learning society. In this observe, we stand for the loss function, it is used to optimize an agent’s parameter in excess of its knowledge, as an imputational graph, and use traditional evolution to develop a population of the imputational graphs over a set of uncomplicated guidance environments. These outcomes in gradually better RL algorithms and the exposed algorithms simplify to more multifaceted environments, even though with visual annotations.</p>

RECOGNITION OF THE TEXT BY MEANS OF DEEP LEARNING

2020 ◽  
Vol 69 (1) ◽  
pp. 378-383
Author(s):  
T.A. Nurmukhanov ◽  
◽  
B.S. Daribayev ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Neural Network Model ◽  
Input Data ◽  
Optimization Methods ◽  
Text Recognition ◽  
Activation Functions

Using neural networks, various variations of the classification of objects can be performed. Neural networks are used in many areas of recognition. A big area in this area is text recognition. The paper considers the optimal way to build a network for text recognition, the use of optimal methods for activation functions, and optimizers. Also, the article checked the correctness of text recognition with different optimization methods. This article is devoted to the analysis of convolutional neural networks. In the article, a convolutional neural network model will be trained with a teacher. Teaching with a teacher is a type of training for neural networks in which you provide the input data and the desired result, that is, the student looking at the input data will understand that you need to strive for the result that was provided to him.

Global optimization with deep-learning-based acceleration surrogate for large-scale seismic acoustic-impedance inversion

Geophysics ◽  
2021 ◽  
pp. 1-74
Author(s):  
Zhaoqi Gao ◽  
Wei Yang ◽  
Yajun Tian ◽  
Chuang Li ◽  
Xiudi Jiang ◽  
...  
Keyword(s):  
Global Optimization ◽  
Deep Learning ◽  
Acoustic Impedance ◽  
Large Scale ◽  
Optimization Methods ◽  
Inversion Method ◽  
Local Minima ◽  
Practical Applications ◽  
Global Optimization Methods ◽  
Low Efficiency

Seismic acoustic-impedance (AI) inversion, which estimates the AI of the reservoir from seismic and other geophysical data, is a type of nonlinear inverse problem that faces the local minima issue during optimization. Without requiring an accurate initial model, global optimization methods have the ability to jump out of local minima and search for the optimal global solution. However, the low-efficiency nature of global optimization methods hinders their practical applications, especially in large-scale AI inversion problems (AI inversion with a large number of traces). We propose a new intelligent seismic AI inversion method based on global optimization and deep learning. In this method, global optimization is used to generate datasets for training a deep learning network and it is used to first accelerate and then surrogate global optimization. In other words, for large-scale seismic AI inversion, global optimization only inverts the AI model for a few traces, and the AI models of most traces are obtained by deep learning. The deep learning architecture that we used to map from seismic trace to its corresponding AI model is established based on U-Net. Because the time-consuming global optimization inversion procedure can be avoided for most traces, this method has a significant advantage over conventional global optimization methods in efficiency. To verify the effectiveness of the proposed method, we compare its performance with the conventional global optimization method on 3D synthetic and field data examples. Compared with the conventional method, the proposed method only needs about one-tenth of the computation time to build AI models with better accuracy.

Comparing optimization methods for deep learning in image processing applications

2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Alexander Geng ◽  
Ali Moghiseh ◽  
Claudia Redenbach ◽  
Katja Schladitz
Keyword(s):  
Image Processing ◽  
Deep Learning ◽  
Optimization Methods ◽  
Optimization Method ◽  
Stochastic Gradient Descent ◽  
Style Transfer ◽  
Artistic Style ◽  
Deep Learning Network ◽  
The Individual ◽  
The Impact

Abstract Training a deep learning network requires choosing its weights such that the output minimizes a given loss function. In practice, stochastic gradient descent is frequently used for solving the optimization problem. Several variants of this approach have been suggested in the literature. We study the impact of the choice of the optimization method on the outcome of the learning process at the example of two image processing applications from quite different fields. The first one is artistic style transfer, where the content of one image is combined with the style of another one. The second application is a real world classification task from industry, namely detecting defects in images of air filters. In both cases, clear differences between the results of the individual optimization methods are observed.

Physics-driven deep-learning inversion with application to transient electromagnetics

Geophysics ◽  
2021 ◽  
Vol 86 (3) ◽  
pp. E209-E224
Author(s):  
Daniele Colombo ◽  
Ersan Turkoglu ◽  
Weichang Li ◽  
Ernesto Sandoval-Curiel ◽  
Diego Rovetta
Keyword(s):  
Deep Learning ◽  
Inverse Problems ◽  
Least Squares ◽  
Optimization Methods ◽  
Optimization Techniques ◽  
Practical Implementation ◽  
Coupling Mechanism ◽  
Transient Electromagnetic ◽  
Model Misfit ◽  
Inversion Techniques

Machine learning, and specifically deep-learning (DL) techniques applied to geophysical inverse problems, is an attractive subject, which has promising potential and, at the same time, presents some challenges in practical implementation. Some obstacles relate to scarce knowledge of the searched geologic structures, a problem that can limit the interpretability and generalizability of the trained DL networks when applied to independent scenarios in real applications. Commonly used (physics-driven) least-squares optimization methods are very efficient local optimization techniques but require good starting models close to the correct solution to avoid local minima. We have developed a hybrid workflow that combines both approaches in a coupled physics-driven/DL inversion scheme. We exploit the benefits and characteristics of both inversion techniques to converge to solutions that typically outperform individual inversion results and bring the solution closer to the global minimum of a nonconvex inverse problem. The completely data-driven and self-feeding procedure relies on a coupling mechanism between the two inversion schemes taking the form of penalty functions applied to the model term. Predictions from the DL network are used to constrain the least-squares inversion, whereas the feedback loop from inversion to the DL scheme consists of the network retraining with partial results obtained from inversion. The self-feeding process tends to converge to a common agreeable solution, which is the result of two independent schemes with different mathematical formalisms and different objective functions on the data and model misfit. We determine that the hybrid procedure is converging to robust and high-resolution resistivity models when applied to the inversion of the synthetic and field transient electromagnetic data. Finally, we speculate that the procedure may be adopted to recast the way we solve inverse problems in several different disciplines.

scholarly journals HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

Sensors ◽  
2021 ◽  
Vol 21 (12) ◽  
pp. 4054
Author(s):  
Kyung-Soo Kim ◽  
Yong-Suk Choi
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Robust Optimization ◽  
Image Classification ◽  
Large Scale ◽  
Optimization Methods ◽  
Optimization Method ◽  
Hybrid Optimization ◽  
Computation Method ◽  
Convolution Neural Networks

As the performance of devices that conduct large-scale computations has been rapidly improved, various deep learning models have been successfully utilized in various applications. Particularly, convolution neural networks (CNN) have shown remarkable performance in image processing tasks such as image classification and segmentation. Accordingly, more stable and robust optimization methods are required to effectively train them. However, the traditional optimizers used in deep learning still have unsatisfactory training performance for the models with many layers and weights. Accordingly, in this paper, we propose a new Adam-based hybrid optimization method called HyAdamC for training CNNs effectively. HyAdamC uses three new velocity control functions to adjust its search strength carefully in term of initial, short, and long-term velocities. Moreover, HyAdamC utilizes an adaptive coefficient computation method to prevent that a search direction determined by the first momentum is distorted by any outlier gradients. Then, these are combined into one hybrid method. In our experiments, HyAdamC showed not only notable test accuracies but also significantly stable and robust optimization abilities when training various CNN models. Furthermore, we also found that HyAdamC could be applied into not only image classification and image segmentation tasks.

Sign in / Sign up

Export Citation Format

Share Document