A Multi-label Classification on Topics of Quranic Verses (English Translation) Using Backpropagation Neural Network with Stochastic Gradient Descent and Adam Optimizer

2019 ◽  
Author(s):  
Nanang Saiful Huda ◽  
Mohamad Syahrul Mubarok ◽  
Adiwijaya
Keyword(s):  
Neural Network ◽  
English Translation ◽  
Gradient Descent ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Backpropagation Neural Network

scholarly journals Neural ODEs as the deep limit of ResNets with constant weights

2020 ◽  
pp. 1-41 ◽  
Author(s):  
Benny Avelin ◽  
Kaj Nyström
Keyword(s):  
Neural Network ◽  
Gradient Descent ◽  
Deep Neural Network ◽  
Theoretical Foundation ◽  
Stochastic Gradient ◽  
Loss Functions ◽  
Stochastic Gradient Descent ◽  
Decay Estimates ◽  
Value Loss ◽  
Fokker Planck Equations

In this paper, we prove that, in the deep limit, the stochastic gradient descent on a ResNet type deep neural network, where each layer shares the same weight matrix, converges to the stochastic gradient descent for a Neural ODE and that the corresponding value/loss functions converge. Our result gives, in the context of minimization by stochastic gradient descent, a theoretical foundation for considering Neural ODEs as the deep limit of ResNets. Our proof is based on certain decay estimates for associated Fokker–Planck equations.

scholarly journals Deep Convolutional Spiking Neural Networks for Image Classification

2021 ◽  
Author(s):  
Ruthvik Vaila
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Gradient Descent ◽  
Stochastic Gradient ◽  
Spiking Neural Networks ◽  
Stochastic Gradient Descent ◽  
Data Set ◽  
Learning Capabilities ◽  
Artificial Neural

Spiking neural networks are biologically plausible counterparts of artificial neural networks. Artificial neural networks are usually trained with stochastic gradient descent (SGD) and spiking neural networks are trained with bioinspired spike timing dependent plasticity (STDP). Spiking networks could potentially help in reducing power usage owing to their binary activations. In this work, we use unsupervised STDP in the feature extraction layers of a neural network with instantaneous neurons to extract meaningful features. The extracted binary feature vectors are then classified using classification layers containing neurons with binary activations. Gradient descent (backpropagation) is used only on the output layer to perform training for classification. Surrogate gradients are proposed to perform backpropagation with binary gradients. The accuracies obtained for MNIST and the balanced EMNIST data set compare favorably with other approaches. The effect of the stochastic gradient descent (SGD) approximations on learning capabilities of our network are also explored. We also studied catastrophic forgetting and its effect on spiking neural networks (SNNs). For the experiments regarding catastrophic forgetting, in the classification sections of the network we use a modified synaptic intelligence that we refer to as cost per synapse metric as a regularizer to immunize the network against catastrophic forgetting in a Single-Incremental-Task scenario (SIT). In catastrophic forgetting experiments, we use MNIST and EMNIST handwritten digits datasets that were divided into five and ten incremental subtasks respectively. We also examine behavior of the spiking neural network and empirically study the effect of various hyperparameters on its learning capabilities using the software tool SPYKEFLOW that we developed. We employ MNIST, EMNIST and NMNIST data sets to produce our results.

scholarly journals Kinerja Algoritma Optimasi Root-Mean-Square Propagation dan Stochastic Gradient Descent pada Klasifikasi Pneumonia Covid-19 Menggunakan CNN

2021 ◽  
Vol 7 (3) ◽  
pp. 420
Author(s):  
Budi Nugroho ◽  
Eva Yulia Puspaningrum ◽  
M. Syahrul Munir
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Root Mean Square ◽  
Gradient Descent ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Mean Square ◽  
X Ray

Penelitian ini berkaitan dengan proses klasifikasi Pneumonia Covid-19 (radang paru-paru atau pneumonia yang disebabkan oleh virus corona SARS-CoV-2) dari citra hasil foto rontgen / x-ray paru-paru dengan menggunakan pendekatan pembelajaran mesin. Klasifikasi dilakukan untuk menentukan apakah kondisi paru-paru seseorang mengalami Pneumonia Covid-19, Pneumonia biasa, atau Normal / Sehat. Untuk menghasilkan kinerja klasifikasi yang lebih baik, proses optimasi seringkali digunakan pada tahap pelatihan data. Banyak teknik yang digunakan untuk melakukan optimasi tersebut, diantaranya adalah algoritma Root-Mean-Square Propagation (RMSprop) dan Stochastic Gradient Descent (SGD). Pada penelitian ini, pengujian dilakukan terhadap kedua metode tersebut untuk mengetahui kinerjanya pada klasifikasi Pneumonia Covid-19. Metode klasifikasi menggunakan Convolutional Neural Network (CNN) yang menerapkan 5 layer konvolusi dengan nilai filter 16, 32, 64, 128, dan 256. Proses pelatihan menggunakan 3.900 citra yang terdiri atas 1.300 citra pneumonia covid-19, 1.300 citra pneumonia, dan 1.300 citra normal. Sedangkan proses validasi menggunakan 450 citra dan proses pengujian mengunakan 225 citra. Berdasarkan uji coba yang telah dilakukan, implementasi algoritma optimasi RMSprop menghasilkan akurasi 87,99%, presisi 0,88, recall 0,86, dan f1 score 0,87. Sedangkan implementasi algoritma optimasi SGD menghasilkan akurasi 66,22%, presisi 0,69, recall 0,64, dan f1 score 0,67. Hasil ini memberikan informasi penting bahwa algoritma optimasi RMSprop menghasilkan kinerja yang jauh lebih baik daripada SGD pada klasifikasi Pneumonia Covid-19.

scholarly journals Multiclass Recognition of Offline Handwritten Devanagari Characters using CNN

2020 ◽  
Vol 5 (6) ◽  
pp. 1429-1439
Author(s):  
Mamta Bisht ◽  
Richa Gupta
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Character Recognition ◽  
Gradient Descent ◽  
Recognition Task ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Handwritten Character Recognition ◽  
Handwritten Documents ◽  
Acceptable Accuracy

The handwriting style of every writer consists of variations, skewness and slanting nature and therefore, it is a stimulating task to recognise these handwritten documents. This article presents a study on various methods available in literature for Devanagari handwritten character recognition and performs its implementation using Convolutional neural network (CNN). Available methods are studied on different parameters and a tabular comparison is also presented which concludes superiority of CNN model in character recognition task. The proposed CNN model results in well acceptable accuracy using dropout and stochastic gradient descent (SGD) optimizer.

Sign in / Sign up

Export Citation Format

Share Document