scholarly journals Optical Recognition of Handwritten Logic Formulas Using Neural Networks

Electronics ◽  
2021 ◽  
Vol 10 (22) ◽  
pp. 2761
Author(s):  
Vaios Ampelakiotis ◽  
Isidoros Perikos ◽  
Ioannis Hatzilygeroudis ◽  
George Tsihrintzis
Keyword(s):  
Neural Networks ◽  
Character Recognition ◽  
Gradient Descent ◽  
Feedforward Neural Networks ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Training Algorithms ◽  
Gradient Descent Algorithm ◽  
Two Stages ◽  
And Training

In this paper, we present a handwritten character recognition (HCR) system that aims to recognize first-order logic handwritten formulas and create editable text files of the recognized formulas. Dense feedforward neural networks (NNs) are utilized, and their performance is examined under various training conditions and methods. More specifically, after three training algorithms (backpropagation, resilient propagation and stochastic gradient descent) had been tested, we created and trained an NN with the stochastic gradient descent algorithm, optimized by the Adam update rule, which was proved to be the best, using a trainset of 16,750 handwritten image samples of 28 × 28 each and a testset of 7947 samples. The final accuracy achieved is 90.13%. The general methodology followed consists of two stages: the image processing and the NN design and training. Finally, an application has been created that implements the methodology and automatically recognizes handwritten logic formulas. An interesting feature of the application is that it allows for creating new, user-oriented training sets and parameter settings, and thus new NN models.

scholarly journals A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization

2021 ◽  
Author(s):  
Tianyi Liu ◽  
Zhehui Chen ◽  
Enlu Zhou ◽  
Tuo Zhao
Keyword(s):  
Neural Networks ◽  
Nonconvex Optimization ◽  
Gradient Descent ◽  
Deep Neural Networks ◽  
Optimization Problems ◽  
Saddle Points ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Nonconvex Optimization Problems ◽  
Empirical Success

Momentum stochastic gradient descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning (e.g., training deep neural networks, variational Bayesian inference, etc.). Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points but hurts the convergence within the neighborhood of optima (if without the step size annealing or momentum annealing). Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.

MapReduce and Optimized Deep Network for Rainfall Prediction in Agriculture

2020 ◽  
Vol 63 (6) ◽  
pp. 900-912
Author(s):  
Oswalt Manoj S ◽  
Ananth J P
Keyword(s):  
Deep Learning ◽  
Gradient Descent ◽  
Prediction Models ◽  
Short Term Memory ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Mean Square ◽  
Rainfall Prediction ◽  
Gradient Descent Algorithm ◽  
Major Factors

Abstract Rainfall prediction is the active area of research as it enables the farmers to move with the effective decision-making regarding agriculture in both cultivation and irrigation. The existing prediction models are scary as the prediction of rainfall depended on three major factors including the humidity, rainfall and rainfall recorded in the previous years, which resulted in huge time consumption and leveraged huge computational efforts associated with the analysis. Thus, this paper introduces the rainfall prediction model based on the deep learning network, convolutional long short-term memory (convLSTM) system, which promises a prediction based on the spatial-temporal patterns. The weights of the convLSTM are tuned optimally using the proposed Salp-stochastic gradient descent algorithm (S-SGD), which is the integration of Salp swarm algorithm (SSA) in the stochastic gradient descent (SGD) algorithm in order to facilitate the global optimal tuning of the weights and to assure a better prediction accuracy. On the other hand, the proposed deep learning framework is built in the MapReduce framework that enables the effective handling of the big data. The analysis using the rainfall prediction database reveals that the proposed model acquired the minimal mean square error (MSE) and percentage root mean square difference (PRD) of 0.001 and 0.0021.

scholarly journals Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks

2020 ◽  
Vol 476 (2239) ◽  
pp. 20200334 ◽  
Author(s):  
Ameya D. Jagtap ◽  
Kenji Kawaguchi ◽  
George Em Karniadakis
Keyword(s):  
Neural Networks ◽  
Adaptive Learning ◽  
Gradient Descent ◽  
Activation Function ◽  
Stochastic Gradient Descent ◽  
Activation Functions ◽  
Gradient Descent Algorithm ◽  
Locally Adaptive ◽  
The Matrix ◽  
Base Method

We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope-based slope recovery term is added in the loss function, which further accelerates convergence, thereby reducing the training cost. On the theoretical side, we prove that in the proposed method, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate, and that the gradient dynamics of the proposed method is not achievable by base methods with any (adaptive) learning rates. We further show that the adaptive activation methods accelerate the convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix–vector product. The different adaptive activation functions are shown to induce different implicit conditioning matrices. Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.

LEAST SQUARE REGRESSION WITH COEFFICIENT REGULARIZATION BY GRADIENT DESCENT

2012 ◽  
Vol 10 (01) ◽  
pp. 1250005 ◽  
Author(s):  
JUAN HUANG ◽  
HONG CHEN ◽  
LUOQING LI
Keyword(s):  
Integral Operator ◽  
Explicit Expression ◽  
Gradient Descent ◽  
Stochastic Gradient ◽  
Least Square ◽  
Stochastic Gradient Descent ◽  
Least Square Regression ◽  
Learning Rates ◽  
Gradient Descent Algorithm ◽  
Regularization Parameters

We propose a stochastic gradient descent algorithm for the least square regression with coefficient regularization. An explicit expression of the solution via sampling operator and empirical integral operator is derived. Learning rates are given in terms of the suitable choices of the step sizes and regularization parameters.

scholarly journals Stochastic Subgradient for Large-Scale Support Vector Machine Using the Generalized Pinball Loss Function

Symmetry ◽  
2021 ◽  
Vol 13 (9) ◽  
pp. 1652
Author(s):  
Wanida Panup ◽  
Rabian Wangkeeree
Keyword(s):  
Support Vector Machine ◽  
Loss Function ◽  
Gradient Descent ◽  
Large Scale ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Support Vector ◽  
Hinge Loss ◽  
Gradient Descent Algorithm ◽  
Pinball Loss

In this paper, we propose a stochastic gradient descent algorithm, called stochastic gradient descent method-based generalized pinball support vector machine (SG-GPSVM), to solve data classification problems. This approach was developed by replacing the hinge loss function in the conventional support vector machine (SVM) with a generalized pinball loss function. We show that SG-GPSVM is convergent and that it approximates the conventional generalized pinball support vector machine (GPSVM). Further, the symmetric kernel method was adopted to evaluate the performance of SG-GPSVM as a nonlinear classifier. Our suggested algorithm surpasses existing methods in terms of noise insensitivity, resampling stability, and accuracy for large-scale data scenarios, according to the experimental results.

Sign in / Sign up

Export Citation Format

Share Document