scholarly journals Learning Rate Optimization in CNN for Accurate Ophthalmic Classification

2021 ◽  
Vol 10 (4) ◽  
pp. 211-216
Author(s):  
Mahmoud Smaida ◽  
Serhii Yaroshchak ◽  
Ahmed Y. Ben Sasi
Keyword(s):  
Deep Learning ◽  
Adaptive Learning ◽  
Optimization Techniques ◽  
Learning Rate ◽  
Training Set ◽  
Model Accuracy ◽  
Adaptive Learning Rate ◽  
Rate Method ◽  
Model Training ◽  
Rate Optimization

One of the most important hyper-parameters for model training and generalization is the learning rate. Recently, many research studies have shown that optimizing the learning rate schedule is very useful for training deep neural networks to get accurate and efficient results. In this paper, different learning rate schedules using some comprehensive optimization techniques have been compared in order to measure the accuracy of a convolutional neural network CNN model to classify four ophthalmic conditions. In this work, a deep learning CNN based on Keras and TensorFlow has been deployed using Python on a database that contains 1692 images, which consists of four types of ophthalmic cases: Glaucoma, Myopia, Diabetic retinopathy, and Normal eyes. The CNN model has been trained on Google Colab. GPU with different learning rate schedules and adaptive learning algorithms. Constant learning rate, time-based decay, step-based decay, exponential decay, and adaptive learning rate optimization techniques for deep learning have been addressed. Adam adaptive learning rate method. has outperformed the other optimization techniques and achieved the best model accuracy of 92.58% for training set and 80.49% for validation datasets, respectively.

MEMS Inertial Sensor Fault Diagnosis Using a CNN-Based Data-Driven Method

2020 ◽  
Vol 34 (14) ◽  
pp. 2059048
Author(s):  
Tong Gao ◽  
Wei Sheng ◽  
Mingliang Zhou ◽  
Bin Fang ◽  
Liping Zheng
Keyword(s):  
Fault Diagnosis ◽  
Adaptive Learning ◽  
Inertial Sensors ◽  
Inertial Sensor ◽  
Optimization Method ◽  
Learning Rate ◽  
Data Driven ◽  
Adaptive Learning Rate ◽  
Improved Performance ◽  
Rate Optimization

In this paper, we propose a novel fault diagnosis (FD) approach for micro-electromechanical systems (MEMS) inertial sensors that recognize the fault patterns of MEMS inertial sensors in an end-to-end manner. We use a convolutional neural network (CNN)-based data-driven method to classify the temperature-related sensor faults in unmanned aerial vehicles (UAVs). First, we formulate the FD problem for MEMS inertial sensors into a deep learning framework. Second, we design a multi-scale CNN which uses the raw data of MEMS inertial sensors as input and which outputs classification results indicating faults. Then we extract fault features in the temperature domain to solve the non-uniform sampling problem. Finally, we propose an improved adaptive learning rate optimization method which accelerates the loss convergence by using the Kalman filter (KF) to train the network efficiently with a small dataset. Our experimental results show that our method achieved high fault recognition accuracy and that our proposed adaptive learning rate method improved performance in terms of loss convergence and robustness on a small training batch.

scholarly journals Plant Diseases Identification through a Discount Momentum Optimizer in Deep Learning

2021 ◽  
Vol 11 (20) ◽  
pp. 9468
Author(s):  
Yunyun Sun ◽  
Yutong Liu ◽  
Haocheng Zhou ◽  
Huijuan Hu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Adaptive Learning ◽  
Learning Rate ◽  
Plant Diseases ◽  
Stochastic Gradient Descent ◽  
Automatic Identification ◽  
Deep Convolutional Neural Networks ◽  
Adaptive Learning Rate

Deep learning proves its promising results in various domains. The automatic identification of plant diseases with deep convolutional neural networks attracts a lot of attention at present. This article extends stochastic gradient descent momentum optimizer and presents a discount momentum (DM) deep learning optimizer for plant diseases identification. To examine the recognition and generalization capability of the DM optimizer, we discuss the hyper-parameter tuning and convolutional neural networks models across the plantvillage dataset. We further conduct comparison experiments on popular non-adaptive learning rate methods. The proposed approach achieves an average validation accuracy of no less than 97% for plant diseases prediction on several state-of-the-art deep learning models and holds a low sensitivity to hyper-parameter settings. Experimental results demonstrate that the DM method can bring a higher identification performance, while still maintaining a competitive performance over other non-adaptive learning rate methods in terms of both training speed and generalization.

An Adaptive Learning Rate Method for Improving Adaptability of Background Models

2013 ◽  
Vol 20 (12) ◽  
pp. 1266-1269 ◽  
Author(s):  
Rui Zhang ◽  
Weiguo Gong ◽  
Victor Grzeda ◽  
Andrew Yaworski ◽  
Michael Greenspan
Keyword(s):  
Adaptive Learning ◽  
Learning Rate ◽  
Adaptive Learning Rate ◽  
Rate Method ◽  
Background Models

scholarly journals Training Deep Neural Networks Using Conjugate Gradient-like Methods

Electronics ◽  
2020 ◽  
Vol 9 (11) ◽  
pp. 1809
Author(s):  
Hideaki Iiduka ◽  
Yu Kobayashi
Keyword(s):  
Neural Networks ◽  
Nonconvex Optimization ◽  
Adaptive Learning ◽  
Stationary Point ◽  
Optimization Problem ◽  
Deep Neural Networks ◽  
Optimization Algorithms ◽  
Learning Rate ◽  
Adaptive Learning Rate ◽  
Rate Optimization

The goal of this article is to train deep neural networks that accelerate useful adaptive learning rate optimization algorithms such as AdaGrad, RMSProp, Adam, and AMSGrad. To reach this goal, we devise an iterative algorithm combining the existing adaptive learning rate optimization algorithms with conjugate gradient-like methods, which are useful for constrained optimization. Convergence analyses show that the proposed algorithm with a small constant learning rate approximates a stationary point of a nonconvex optimization problem in deep learning. Furthermore, it is shown that the proposed algorithm with diminishing learning rates converges to a stationary point of the nonconvex optimization problem. The convergence and performance of the algorithm are demonstrated through numerical comparisons with the existing adaptive learning rate optimization algorithms for image and text classification. The numerical results show that the proposed algorithm with a constant learning rate is superior for training neural networks.

Sign in / Sign up

Export Citation Format

Share Document