scholarly journals Entropy Learning in Neural Network

2017 ◽  
Vol 20 (3&4) ◽  
pp. 307-322
Author(s):  
Geok See Ng ◽  
D. Shi ◽  
A. Wahab ◽  
H. Singh
Keyword(s):  
Neural Network ◽  
Learning Phase ◽  
Entropy Term ◽  
The Neural Network ◽  
Important Nodes ◽  
Hidden Nodes

In this paper, entropy term is used in the learning phase of a neural network.  As learning progresses, more hidden nodes get into saturation.  The early creation of such hidden nodes may impair generalisation.  Hence entropy approach is proposed to dampen the early creation of such nodes.  The entropy learning also helps to increase the importance of relevant nodes while dampening the less important nodes.  At the end of learning, the less important nodes can then be eliminated to reduce the memory requirements of the neural network.

ENTROPY LEARNING AND RELEVANCE CRITERIA FOR NEURAL NETWORK PRUNING

2003 ◽  
Vol 13 (05) ◽  
pp. 291-305 ◽  
Author(s):  
GEOK SEE NG ◽  
ABDUL WAHAB ◽  
DAMING SHI
Keyword(s):  
Neural Network ◽  
Learning Phase ◽  
Network Pruning ◽  
The Neural Network ◽  
Relevance Criteria ◽  
Important Nodes ◽  
Hidden Nodes

In this paper, entropy is a term used in the learning phase of a neural network. As learning progresses, more hidden nodes get into saturation. The early creation of such hidden nodes may impair generalisation. Hence an entropy approach is proposed to dampen the early creation of such nodes by using a new computation called entropy cycle. Entropy learning also helps to increase the importance of relevant nodes while dampening the less important nodes. At the end of learning, the less important nodes can then be pruned to reduce the memory requirements of the neural network.

scholarly journals Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

2020 ◽  
Author(s):  
Lifu Wang ◽  
Bo Shen ◽  
Ning Zhao ◽  
Zhiyuan Zhang
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Level Sets ◽  
Theoretical Explanation ◽  
Learning Ability ◽  
Local Minima ◽  
Global Minima ◽  
Residual Network ◽  
The Neural Network ◽  
Hidden Nodes

The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to “guarantee" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works [Freeman and Bruna, 2016], it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number m of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very “shallow". The “depth" of these local minima are at most O(m^(η-1)/n), where n is the input dimension, η<1. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.

scholarly journals Study of Variants of Extreme Learning Machine (ELM) Brands and its Performance Measure on Classification Algorithm

2021 ◽  
Vol 3 (2) ◽  
pp. 83-95
Keyword(s):  
Neural Network ◽  
Computation Time ◽  
Performance Measure ◽  
Feed Forward Neural Network ◽  
Forward Algorithm ◽  
Feed Forward ◽  
Research Article ◽  
The Neural Network ◽  
Real Essence ◽  
Hidden Nodes

Recently, the feed-forward neural network is functioning with slow computation time and increased gain. The weight vector and biases in the neural network can be tuned based on performing intelligent assignment for simple generalized operation. This drawback of FFNN is solved by using various ELM algorithms based on the applications issues. ELM algorithms have redesigned the existing neural networks with network components such as hidden nodes, weights, and biases. The selection of hidden nodes is randomly determined and leverages good accuracy than conservative methods. The main aim of this research article is to explain variants of ELM advances for different applications. This procedure can be improved and optimized by using the neural network with novel feed-forward algorithm. The nodes will mainly perform due to the above factors, which are tuning for inverse operation. The ELM essence should be incorporated to reach a faster learning speed and less computation time with minimum human intervention. This research article consists of the real essence of ELM and a briefly explained algorithm for classification purpose. This research article provides clear information on the variants of ELM for different classification tasks. Finally, this research article has discussed the future extension of ELM for several applications based on the function approximation.

scholarly journals Predicting the Earthquake Magnitude Using the Multilayer Perceptron Neural Network with Two Hidden Layers

2016 ◽  
Vol 2 (1) ◽  
pp. 1-12 ◽  
Author(s):  
Jamal Mahmoudi ◽  
Mohammad Ali Arjomand ◽  
Masoud Rezaei ◽  
Mohammad Hossein Mohammadi
Keyword(s):  
Neural Network ◽  
Multilayer Perceptron ◽  
Current Method ◽  
Real Data ◽  
Good Choice ◽  
Mlp Neural Network ◽  
The Neural Network ◽  
Input Layer ◽  
Hidden Layer ◽  
Hidden Nodes

Because of the major disadvantages of previous methods for calculating the magnitude of the earthquakes, the neural network as a new method is examined. In this paper a kind of neural network named Multilayer Perceptron (MLP) is used to predict magnitude of earthquakes. MLP neural network consist of three main layers; input layer, hidden layer and output layer. Since the best network configurations such as the best number of hidden nodes and the most appropriate training method cannot be determined in advance, and also, overtraining is possible, 128 models of network are evaluated to determine the best prediction model. By comparing the results of the current method with the real data, it can be concluded that MLP neural network has high ability in predicting the magnitude of earthquakes and it’s a very good choice for this purpose.

scholarly journals Detection of Signature Based Forgeries Using Artificial Neural Network

2019 ◽  
Vol 9 (1S) ◽  
pp. 400-403
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Human Brain ◽  
The Neural Network ◽  
Artificial Neural ◽  
Hidden Nodes

Signature plays an important role in banking, financial, commercial etc. Signature may be unique for each person. In olden days, no techniques were used to find the forged signature and it becomes a tremendous strain for human brain. Sometimes the forged signature may also believe as an original one. But nowadays, there are so many methods to detect the forged signature. This paper explains about identifying the forged signature from original signature. The signatures are preprocessed then the features such area, centroid coordinate, eccentricity, kurtosis are extracted. Then it is classified using Artificial Neural Network effectively. The result is analyzed by changing the hidden nodes present in the Neural Network. The performance is evaluated using the parameters such as TPR, TNR, FPR and FNR.

Neural Network for Automatic Analysis of Motility Data

1994 ◽  
Vol 33 (01) ◽  
pp. 157-160 ◽  
Author(s):  
S. Kruse-Andersen ◽  
J. Kolberg ◽  
E. Jakobsen
Keyword(s):  
Neural Network ◽  
Muscular Activity ◽  
Recording System ◽  
Automatic Analysis ◽  
Biologically Relevant ◽  
System A ◽  
The Neural Network ◽  
Valuable Method ◽  
Motor Abnormalities ◽  
Trained Network

Abstract:Continuous recording of intraluminal pressures for extended periods of time is currently regarded as a valuable method for detection of esophageal motor abnormalities. A subsequent automatic analysis of the resulting motility data relies on strict mathematical criteria for recognition of pressure events. Due to great variation in events, this method often fails to detect biologically relevant pressure variations. We have tried to develop a new concept for recognition of pressure events based on a neural network. Pressures were recorded for over 23 hours in 29 normal volunteers by means of a portable data recording system. A number of pressure events and non-events were selected from 9 recordings and used for training the network. The performance of the trained network was then verified on recordings from the remaining 20 volunteers. The accuracy and sensitivity of the two systems were comparable. However, the neural network recognized pressure peaks clearly generated by muscular activity that had escaped detection by the conventional program. In conclusion, we believe that neu-rocomputing has potential advantages for automatic analysis of gastrointestinal motility data.

Decision Support for Psychiatric Diagnosis Based on a Simple Questionnaire

1997 ◽  
Vol 36 (04/05) ◽  
pp. 349-351
Author(s):  
H. Mizuta ◽  
K. Kawachi ◽  
H. Yoshida ◽  
K. Iida ◽  
Y. Okubo ◽  
...  
Keyword(s):  
Neural Network ◽  
Decision Support ◽  
Psychiatric Diagnosis ◽  
Psychiatric Patients ◽  
Bayesian Classifier ◽  
Neural Network Classifier ◽  
Correct Decision ◽  
Neurotic Disorders ◽  
The Neural Network

Abstract:This paper compares two classifiers: Pseudo Bayesian and Neural Network for assisting in making diagnoses of psychiatric patients based on a simple yes/no questionnaire which is provided at the outpatient’s first visit to the hospital. The classifiers categorize patients into three most commonly seen ICD classes, i.e. schizophrenic, emotional and neurotic disorders. One hundred completed questionnaires were utilized for constructing and evaluating the classifiers. Average correct decision rates were 73.3% for the Pseudo Bayesian Classifier and 77.3% for the Neural Network classifier. These rates were higher than the rate which an experienced psychiatrist achieved based on the same restricted data as the classifiers utilized. These classifiers may be effectively utilized for assisting psychiatrists in making their final diagnoses.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

The System for Speech Recognition on the Basis of the Neural Network

2004 ◽  
Vol 62 (1-6) ◽  
pp. 131-142
Author(s):  
V. A. Pimenov
Keyword(s):  
Neural Network ◽  
Speech Recognition ◽  
The Neural Network
Sign in / Sign up

Export Citation Format

Share Document