Neural networks and Sigmoid Activation Function in Multi-Layer Networks

2020 ◽  
Vol 1 (2) ◽  
pp. 29-43
Author(s):  
Renas M Redwan
Keyword(s):  
Neural Networks ◽  
Logic Gate ◽  
Back Propagation ◽  
Main Idea ◽  
Activation Function ◽  
Back Propagation Neural Networks ◽  
Linear Threshold ◽  
Non Linear ◽  
Hidden Layer ◽  
Sigmoid Activation Function

Back propagation neural networks are known for computing the problems that cannot easily be computed (huge datasets analysis or training) in artificial neural networks. The main idea of this paper is to implement XOR logic gate by ANNs using back propagation neural networks for back propagation of errors, and sigmoid activation function. This neural networks to map non-linear threshold gate. The non-linear used to classify binary inputs ( ) and passing it through hidden layer for computing  and  ( ), after computing errors by ( ) the weights and thetas ( ) are changing according to errors. Sigmoid activation function is =  and Derivation of sigmoid is = . The sig(x) and Dsig(x) is between 1 to 0.

scholarly journals Back Propagation Neural Network(BPNN) and Sigmoid Activation Function in Multi-Layer Networks

2019 ◽  
Vol 8 (4) ◽  
pp. 216
Author(s):  
Renas Rajab Asaad ◽  
Rasan I. Ali
Keyword(s):  
Neural Network ◽  
Logic Gate ◽  
Back Propagation ◽  
Main Idea ◽  
Activation Function ◽  
Back Propagation Neural Network ◽  
Linear Threshold ◽  
Non Linear ◽  
Hidden Layer ◽  
Sigmoid Activation Function

Back propagation neural network are known for computing the problems that cannot easily be computed (huge datasets analysis or training) in artificial neural networks. The main idea of this paper is to implement XOR logic gate by ANNs using back propagation neural network for back propagation of errors, and sigmoid activation function. This neural network to map non-linear threshold gate. The non-linear used to classify binary inputs (x1, x2) and passing it through hidden layer for computing coefficient_errors and gradient_errors (Cerrors, Gerrors), after computing errors by (ei = Output_desired- Output_actual) the weights and thetas (ΔWji = (α)(Xj)(gi), Δϴj = (α)(-1)(gi)) are changing according to errors. Sigmoid activation function is = sig(x)=1/(1+e-x) and Derivation of sigmoid is = dsig(x) = sig(x)(1-sig(x)). The sig(x) and Dsig(x) is between 1 to 0.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

NEW METHOD OF NEURON DESIGN BASED ON DISCRETE Z-FUNCTION TO ADAPT THE CHANGE OF INTEGRATED VEHICLE STABILITY CONTROL ORDER

2009 ◽  
Vol 08 (03) ◽  
pp. 253-285 ◽  
Author(s):  
M. HARLY ◽  
I. N. SUTANTRA ◽  
H. P. MAURIDHI
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Back Propagation ◽  
Activation Function ◽  
Zero Order ◽  
Vehicle Stability ◽  
Adaptive Control System ◽  
Control Performance ◽  
Fixed Order ◽  
Hidden Layer

Fixed order neural networks (FONN), such as high order neural network (HONN), in which its architecture is developed from zero order of activation function and joint weight, regulates only the number of weight and their value. As a result, this network only produces a fixed order model or control level. These obstacles, which affect preceeding architectures, have been performing finite ability to adapt uncertainty character of real world plant, such as driving dynamics and its desired control performance. This paper introduces a new concept of neural network neuron. In this matter, exploiting discrete z-function builds new neuron activation. Instead of zero order joint weight matrices, the discrete z-function weight matrix will be provided to realize uncertainty or undetermined real word plant and desired adaptive control system that their order has probably been changing. Instead of using bias, an initial condition value is developed. Neural networks using new neurons is called Varied Order Neural Network (VONN). For optimization process, updating order, coefficient and initial value of node activation function uses GA; while updating joint weight, it applies both back propagation (combined LSE-gauss Newton) and NPSO. To estimate the number of hidden layer, constructive back propagation (CBP) was also applied. Thorough simulation was conducted to compare the control performance between FONN and MONN. In order to control, vehicle stability was equipped by electronics stability program (ESP), electronics four wheel steering (4-EWS), and active suspension (AS). 2000, 4000, 6000, 8000 data that are from TODS, a hidden layer, 3 input nodes, 3 output nodes were provided to train and test the network of both the uncertainty model and its adaptive control system. The result of simulation, therefore, shows that stability parameter such as yaw rate error, vehicle side slip error, and rolling angle error produces better performance control in the form of smaller performance index using FDNN than those using MONN.

Selecting the architecture of a class of back-propagation neural networks used as approximators

1997 ◽  
Vol 11 (1) ◽  
pp. 33-44 ◽  
Author(s):  
William C. Carpenter ◽  
Margery E. Hoffman
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Back Propagation ◽  
Region Of Interest ◽  
Input Output ◽  
Advantages And Disadvantages ◽  
Back Propagation Neural Networks ◽  
Input Layer ◽  
Hidden Layer

AbstractThis paper examines the architecture of back-propagation neural networks used as approximators by addressing the interrelationship between the number of training pairs and the number of input, output, and hidden layer nodes required for a good approximation. It concentrates on nets with an input layer, one hidden layer, and one output layer. It shows that many of the currently proposed schemes for selecting network architecture for such nets are deficient. It demonstrates in numerous examples that overdetermined neural networks tend to give good approximations over a region of interest, while underdetermined networks give approximations which can satisfy the training pairs but may give poor approximations over that region of interest. A scheme is presented that adjusts the number of hidden layer nodes in a neural network so as to give an overdetermined approximation. The advantages and disadvantages of using multiple output nodes are discussed. Guidelines for selecting the number of output nodes are presented.

scholarly journals Prediction of Dengue Hemorrhagic Fever Cases Based on Weather Parameters Using Back Propagation Neural Networks (Case Study in Pontianak City)

2019 ◽  
Vol 15 (2) ◽  
pp. 114-121
Author(s):  
I T Rahayu ◽  
N Nurhasanah ◽  
R Adriat
Keyword(s):  
Neural Networks ◽  
Dengue Hemorrhagic Fever ◽  
Strong Correlation ◽  
Network Architecture ◽  
Back Propagation ◽  
Hemorrhagic Fever ◽  
Activation Function ◽  
Neuron Network ◽  
Back Propagation Neural Networks ◽  
Weather Parameters

Research has been conducted by predicting cases of dengue hemorrhagic fever based on weather parameters. The data used are weather parameters in the form of air temperature data, air humidity, rainfall, duration of solar radiation and wind speed as input data and data on dengue hemorrhagic fever cases as the target data. This study aims to see the confirmation of dengue hemorrhagic parameters in Pontianak. The benefit in the field of education is that students and teachers are aware of the dangers of dengue because it can cause death. The method used is back propagation neural networks with the best network architecture in predicting cases of dengue hemorrhagic fever are [50 40 30 1] and binary sigmoid activation function, bipolar sigmoid and linear function. The activation function will determine whether the signal from the neuron input will be forwarded to other neurons and is also used to determine the output of a neuron. Network training correlation value is 0.9995 (very strong correlation) with MSE 0.0001 and network testing is 0.9325 (very strong correlation) with MSE 1.61. Determination coefficient serve as accuracy with values obtained is 0.85, which means that 85% of weather parameters can be used as input in predicting the incidence of dengue hemorrhagic fever in Pontianak City.

scholarly journals Application of Artificial Neural Networks for estimating index floods

2012 ◽  
Vol 42 (4) ◽  
pp. 295-311 ◽  
Author(s):  
Viliam Šimor ◽  
Kamila Hlavčová ◽  
Silvia Kohnová ◽  
Ján Szolgay
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Multiple Regression ◽  
Regression Models ◽  
Learning Algorithm ◽  
Back Propagation ◽  
Activation Function ◽  
Multiple Regression Models ◽  
Artificial Neural ◽  
Hidden Layer

Abstract This article presents an application of Artificial Neural Networks (ANNs) and multiple regression models for estimating mean annual maximum discharge (index flood) at ungauged sites. Both approaches were tested for 145 small basins in Slovakia in areas ranging from 20 to 300 km2. Using the objective clustering method, the catchments were divided into ten homogeneous pooling groups; for each pooling group, mutually independent predictors (catchment characteristics) were selected for both models. The neural network was applied as a simple multilayer perceptron with one hidden layer and with a back propagation learning algorithm. Hyperbolic tangents were used as an activation function in the hidden layer. Estimating index floods by the multiple regression models were based on deriving relationships between the index floods and catchment predictors. The efficiencies of both approaches were tested by the Nash-Sutcliffe and a correlation coefficients. The results showed the comparative applicability of both models with slightly better results for the index floods achieved using the ANNs methodology.

scholarly journals Non-linear Neurons with Human-like Apical Dendrite Activations

2020 ◽  
Author(s):  
Mariana-Iuliana Georgescu ◽  
Radu Tudor Ionescu ◽  
Nicolae-Catalin Ristea ◽  
Nicu Sebe
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Apical Dendrite ◽  
Activation Function ◽  
Data Sets ◽  
Network Architectures ◽  
Logical Function ◽  
Non Linear ◽  
Hidden Layer ◽  
Decision Boundaries

<pre>In order to classify linearly non-separable data, neurons are typically organized into multi-layer neural networks that are equipped with at least one hidden layer. Inspired by some recent discoveries in neuroscience, we propose a new neuron model along with a novel activation function enabling learning of non-linear decision boundaries using a single neuron. We show that a standard neuron followed by the novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. Furthermore, we conduct experiments on three benchmark data sets from computer vision and natural language processing, i.e. Fashion-MNIST, UTKFace and MOROCO, showing that the ADA and the leaky ADA functions provide superior results to Rectified Liner Units (ReLU) and leaky ReLU, for various neural network architectures, e.g. 1-hidden layer or 2-hidden layers multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs) such as LeNet, VGG, ResNet and Character-level CNN. We also obtain further improvements when we change the standard model of the neuron with our pyramidal neuron with apical dendrite activations (PyNADA).<br></pre>

Guidelines for the selection of network architecture

1997 ◽  
Vol 11 (5) ◽  
pp. 395-408 ◽  
Author(s):  
William C. Carpenter ◽  
Margery E. Hoffman
Keyword(s):  
Neural Networks ◽  
Network Architecture ◽  
Back Propagation ◽  
Test Functions ◽  
Back Propagation Neural Networks ◽  
Numerous Test ◽  
Hidden Layer ◽  
Selection Of

AbstractThis paper is concerned with presenting guidelines to aide in the selection of the appropriate network architecture for back-propagation neural networks used as approximators. In particular, its goal is to indicate under what circumstances neural networks should have two hidden layers and under what circumstances they should have one hidden layer. Networks with one and with two hidden layers were used to approximate numerous test functions. Guidelines were developed from the results of these investigations.

scholarly journals A Non-Polynomial, Non-Sigmoidal, Bounded and Symmetric Activation Function for Feed – Forward Artificial Neural Networks

2019 ◽  
Vol 8 (12) ◽  
pp. 405-410
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Function Approximation ◽  
Activation Function ◽  
Continuous Functions ◽  
Feed Forward ◽  
Learning Tasks ◽  
Non Linear ◽  
Artificial Neural ◽  
Hidden Layer

Feed-forward artificial neural networks are universal approximators of continuous functions. This property enables the use of these networks to solve learning tasks. Learning tasks in this paradigm are cast as function approximation problems. The universal approximation results for these networks require at least one hidden layer with non-linear nodes, and also require that the non-linearities be non-polynomial in nature. In this paper a non-polynomial and non-sigmoidal non-linear function is proposed as a suitable activation function for these networks. The usefulness of the proposed activation function is shown on 12 function approximation task. The obtained results demonstrate that the proposed activation function outperforms the logistic / log-sigmoid and the hyperbolic tangent activation functions.

Sign in / Sign up

Export Citation Format

Share Document