Computational Complexity of Gradient Descent Algorithm

Batch Size ◽

Automated Learning ◽

The World ◽

Different Types ◽

Insight Into

Information is mounting exponentially, and the world is moving to hunt knowledge with the help of Big Data. The labelled data is used for automated learning and data analysis which is termed as Machine Learning. Linear Regression is a statistical method for predictive analysis. Gradient Descent is the process which uses cost function on gradients for minimizing the complexity in computing mean square error. This work presents an insight into the different types of Gradient descent algorithms namely, Batch Gradient Descent, Stochastic Gradient Descent and Mini-Batch Gradient Descent, which are implemented on a Linear regression dataset, and hence determine the computational complexity and other factors like learning rate, batch size and number of iterations which affect the efficiency of the algorithm.

Revisiting the dynamic interactions between economic growth and environmental pollution in Italy: evidence from a gradient descent algorithm

Environmental Science and Pollution Research ◽

10.1007/s11356-021-14264-z ◽

2021 ◽

Author(s):

Marco Mele ◽

Cosimo Magazzino ◽

Nicolas Schneider ◽

Floriana Nicolai

Keyword(s):

Economic Growth ◽

Gradient Descent ◽

Statistical Data ◽

Policy Recommendations ◽

Dynamic Interactions ◽

The Relationship ◽

Main Strand ◽

Yearly Data

AbstractAlthough the literature on the relationship between economic growth and CO2 emissions is extensive, the use of machine learning (ML) tools remains seminal. In this paper, we assess this nexus for Italy using innovative algorithms, with yearly data for the 1960–2017 period. We develop three distinct models: the batch gradient descent (BGD), the stochastic gradient descent (SGD), and the multilayer perceptron (MLP). Despite the phase of low Italian economic growth, results reveal that CO2 emissions increased in the predicting model. Compared to the observed statistical data, the algorithm shows a correlation between low growth and higher CO2 increase, which contradicts the main strand of literature. Based on this outcome, adequate policy recommendations are provided.

Soft-Sign Stochastic Gradient Descent Algorithm for Wireless Federated Learning

10.1109/spawc51858.2021.9593212 ◽

2021 ◽

Author(s):

Seunghoon Lee ◽

Chanho Park ◽

Songnam Hong ◽

Yonina C. Eldar ◽

Namyoon Lee

Keyword(s):

Gradient Descent ◽

Stochastic Gradient ◽

Descent Algorithm ◽

Gradient Descent Algorithm

Optical Recognition of Handwritten Logic Formulas Using Neural Networks

Electronics ◽

10.3390/electronics10222761 ◽

2021 ◽

Vol 10 (22) ◽

pp. 2761

Author(s):

Vaios Ampelakiotis ◽

Isidoros Perikos ◽

Ioannis Hatzilygeroudis ◽

George Tsihrintzis

Keyword(s):

Neural Networks ◽

Character Recognition ◽

Gradient Descent ◽

Feedforward Neural Networks ◽

Stochastic Gradient ◽

Training Algorithms ◽

Two Stages ◽

And Training

In this paper, we present a handwritten character recognition (HCR) system that aims to recognize first-order logic handwritten formulas and create editable text files of the recognized formulas. Dense feedforward neural networks (NNs) are utilized, and their performance is examined under various training conditions and methods. More specifically, after three training algorithms (backpropagation, resilient propagation and stochastic gradient descent) had been tested, we created and trained an NN with the stochastic gradient descent algorithm, optimized by the Adam update rule, which was proved to be the best, using a trainset of 16,750 handwritten image samples of 28 × 28 each and a testset of 7947 samples. The final accuracy achieved is 90.13%. The general methodology followed consists of two stages: the image processing and the NN design and training. Finally, an application has been created that implements the methodology and automatically recognizes handwritten logic formulas. An interesting feature of the application is that it allows for creating new, user-oriented training sets and parameter settings, and thus new NN models.

Electrocardiogram Classification for Arrhythmia using Convolutional Neural Network 2D and Adabound Optimizer

International Journal of Recent Technology and Engineering - 2 ◽

10.35940/ijrte.e4591.018520 ◽

2020 ◽

Vol 8 (5) ◽

pp. 1277-1284

Keyword(s):

Neural Network ◽

Cardiovascular Disease ◽

Convolutional Neural Network ◽

Gradient Descent ◽

Transform Method ◽

The World ◽

Optimal Accuracy ◽

Deadly Disease ◽

Electrocardiogram Ecg

Cardiovascular disease is the number one deadly disease in the world. Arrhythmia is one of the types of cardiovascular disease which is hard to detect but by using the routine electrocardiogram (ECG) recording. Due to the variety and the noise of ECG, it is very time consuming to detect it only by experts using bare eyes.Learning from the previous research in order to help the experts, this research develop 11 layers Convolutional Neural Network 2D (CNN 2D) using MITBIH Arrhythmia Dataset. The dataset is firstly preprocessed by using wavelet transform method, then being segmented by R-peak method. The challenge is how to conquer the imbalance and small amount of data but still get the optimal accuracy. This research can be helpful in helping the doctors figure out the type of arrhythmia of the patient. Therefore, this research did the comparison of various optimizers attach in CNN 2D namely, Adabound, Adadelta, Adagrad, Amsbound, Adam and Stochastic Gradient Descent (SGD). The result is Adabound get the highest performance with 91% accuracy and faster 1s training duration than Adam which is approximately 18s per epoch.

A Stochastic Gradient Descent Algorithm for Structural Risk Minimisation

Lecture Notes in Computer Science - Algorithmic Learning Theory ◽

10.1007/978-3-540-39624-6_17 ◽

2003 ◽

pp. 205-220 ◽

Cited By ~ 1

Author(s):

Joel Ratsaby

Keyword(s):

Gradient Descent ◽

Stochastic Gradient ◽

Risk Minimisation ◽

Descent Algorithm ◽

Structural Risk

MapReduce and Optimized Deep Network for Rainfall Prediction in Agriculture

The Computer Journal ◽

10.1093/comjnl/bxz164 ◽

2020 ◽

Vol 63 (6) ◽

pp. 900-912

Author(s):

Oswalt Manoj S ◽

Ananth J P

Keyword(s):

Deep Learning ◽

Gradient Descent ◽

Prediction Models ◽

Short Term Memory ◽

Stochastic Gradient ◽

Mean Square ◽

Rainfall Prediction ◽

Major Factors

Abstract Rainfall prediction is the active area of research as it enables the farmers to move with the effective decision-making regarding agriculture in both cultivation and irrigation. The existing prediction models are scary as the prediction of rainfall depended on three major factors including the humidity, rainfall and rainfall recorded in the previous years, which resulted in huge time consumption and leveraged huge computational efforts associated with the analysis. Thus, this paper introduces the rainfall prediction model based on the deep learning network, convolutional long short-term memory (convLSTM) system, which promises a prediction based on the spatial-temporal patterns. The weights of the convLSTM are tuned optimally using the proposed Salp-stochastic gradient descent algorithm (S-SGD), which is the integration of Salp swarm algorithm (SSA) in the stochastic gradient descent (SGD) algorithm in order to facilitate the global optimal tuning of the weights and to assure a better prediction accuracy. On the other hand, the proposed deep learning framework is built in the MapReduce framework that enables the effective handling of the big data. The analysis using the rainfall prediction database reveals that the proposed model acquired the minimal mean square error (MSE) and percentage root mean square difference (PRD) of 0.001 and 0.0021.

Proceedings of The Royal Society A Mathematical Physical and Engineering Sciences ◽

Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks

10.1098/rspa.2020.0334 ◽

2020 ◽

Vol 476 (2239) ◽

pp. 20200334 ◽

Cited By ~ 2

Author(s):

Ameya D. Jagtap ◽

Kenji Kawaguchi ◽

George Em Karniadakis

Keyword(s):

Neural Networks ◽

Adaptive Learning ◽

Gradient Descent ◽

Activation Function ◽

Activation Functions ◽

Locally Adaptive ◽

The Matrix ◽

Base Method

We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope-based slope recovery term is added in the loss function, which further accelerates convergence, thereby reducing the training cost. On the theoretical side, we prove that in the proposed method, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate, and that the gradient dynamics of the proposed method is not achievable by base methods with any (adaptive) learning rates. We further show that the adaptive activation methods accelerate the convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix–vector product. The different adaptive activation functions are shown to induce different implicit conditioning matrices. Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.