Wirtinger Calculus Based Gradient Descent and Levenberg-Marquardt Learning Algorithms in Complex-Valued Neural Networks

2011 ◽  
pp. 550-559 ◽  
Author(s):  
Md. Faijul Amin ◽  
Muhammad Ilias Amin ◽  
A. Y. H. Al-Nuaimi ◽  
Kazuyuki Murase
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Learning Algorithms ◽  
Wirtinger Calculus ◽  
Levenberg Marquardt ◽  
Complex Valued

scholarly journals On “Natural” Learning and Pruning in Multilayered Perceptrons

2000 ◽  
Vol 12 (4) ◽  
pp. 881-901 ◽  
Author(s):  
Tom Heskes
Keyword(s):  
Gradient Descent ◽  
Learning Algorithms ◽  
Natural Gradient ◽  
Batch Mode ◽  
Fisher Matrix ◽  
Levenberg Marquardt ◽  
On Line ◽  
On Line Learning ◽  
Optimal Brain Surgeon

Several studies have shown that natural gradient descent for on-line learning is much more efficient than standard gradient descent. In this article, we derive natural gradients in a slightly different manner and discuss implications for batch-mode learning and pruning, linking them to existing algorithms such as Levenberg-Marquardt optimization and optimal brain surgeon. The Fisher matrix plays an important role in all these algorithms. The second half of the article discusses a layered approximation of the Fisher matrix specific to multilayered perceptrons. Using this approximation rather than the exact Fisher matrix, we arrive at much faster “natural” learning algorithms and more robust pruning procedures.

scholarly journals The Eighty Five Percent Rule for Optimal Learning

2018 ◽  
Author(s):  
Robert C. Wilson ◽  
Amitai Shenhav ◽  
Mark Straccia ◽  
Jonathan D. Cohen
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Broad Class ◽  
Binary Classification ◽  
Learning Algorithms ◽  
Sweet Spot ◽  
Optimal Learning ◽  
Rate Of Learning ◽  
Classification Tasks

AbstractResearchers and educators have long wrestled with the question of how best to teach their clients be they human, animal or machine. Here we focus on the role of a single variable, the difficulty of training, and examine its effect on the rate of learning. In many situations we find that there is a sweet spot in which training is neither too easy nor too hard, and where learning progresses most quickly. We derive conditions for this sweet spot for a broad class of learning algorithms in the context of binary classification tasks, in which ambiguous stimuli must be sorted into one of two classes. For all of these gradient-descent based learning algorithms we find that the optimal error rate for training is around 15.87% or, conversely, that the optimal training accuracy is about 85%. We demonstrate the efficacy of this ‘Eighty Five Percent Rule’ for artificial neural networks used in AI and biologically plausible neural networks thought to describe human and animal learning.

scholarly journals Prediction of students’ academic performance using ANN with mini-batch gradient descent and Levenberg-Marquardt optimization algorithms

2021 ◽  
Vol 2106 (1) ◽  
pp. 012018
Author(s):  
F R J Simanungkalit ◽  
H Hanifah ◽  
G Ardaneswari ◽  
N Hariadi ◽  
B D Handari
Keyword(s):  
Neural Networks ◽  
Academic Performance ◽  
Gradient Descent ◽  
Optimization Algorithms ◽  
At Risk Students ◽  
Learning Activity ◽  
Average Sensitivity ◽  
Activity Data ◽  
Average Accuracy ◽  
Levenberg Marquardt

Abstract Online learning indirectly increases stress, thereby reducing social interaction among students and leading to physical and mental fatigue, which in turn reduced students’ academic performance. Therefore, the prediction of academic performance is required sooner to identify at-risk students with declining performance. In this paper, we use artificial neural networks (ANN) to predict this performance. ANNs with two optimization algorithms, mini-batch gradient descent and Levenberg-Marquardt, are implemented on students’ learning activity data in course X, which is recorded on LMS UI. Data contains 232 students and consists of two periods: the first month and second month of study. Before ANNs are implemented, both normalization and usage of ADASYN are conducted. The results of ANN implementation using two optimization algorithms within 10 trials each are compared based on the average accuracy, sensitivity, and specificity values. We then determine the best period to predict unsuccessful students correctly. The results show that both algorithms give better predictions over two months instead of one. ANN with mini-batch gradient descent has an average sensitivity of 78%; the corresponding values for ANN with Levenberg-Marquardt are 75%. Therefore, ANN with mini-batch gradient descent as its optimization algorithm is more suitable for predicting students that have potential to fail.

Sign in / Sign up

Export Citation Format

Share Document