Multilayer Perceptron New Method for Selecting the Architecture Based on the Choice of Different Activation Functions

2019 ◽  
Vol 11 (4) ◽  
pp. 21-34
Author(s):  
Hassan Ramchoun ◽  
Mohammed Amine Janati Idrissi ◽  
Youssef Ghanou ◽  
Mohamed Ettaouil
Keyword(s):  
Multilayer Perceptron ◽  
Network Architecture ◽  
Back Propagation ◽  
Activation Function ◽  
Activation Functions ◽  
Classification Problems ◽  
Function Type ◽  
New Approach ◽  
And Performance ◽  
Selection Of

Multilayer perceptron has a large amount of classifications and regression applications in many fields: pattern recognition, voice, and classification problems. But the architecture choice in particular, the activation function type used for each neuron has a great impact on the convergence and performance. In the present article, the authors introduce a new approach to optimize the selection of network architecture, weights, and activation functions. To solve the obtained model the authors use a genetic algorithm and train the network with a back-propagation method. The numerical results show the effectiveness of the approach shown in this article, and the advantages of the new model compared to the existing previous model in the literature.

Multilayer Perceptron Based Equalizer with an Improved Back Propagation Algorithm for Nonlinear Channels

2016 ◽  
Vol 7 (3) ◽  
pp. 16-31 ◽  
Author(s):  
Zohra Zerdoumi ◽  
Djamel Chikouche ◽  
Djamel Benatia
Keyword(s):  
Multilayer Perceptron ◽  
Additive Noise ◽  
Back Propagation ◽  
Activation Function ◽  
Bp Algorithm ◽  
Decision Feedback ◽  
Back Propagation Algorithm ◽  
New Approach ◽  
Nonlinear Channels ◽  
Training Efficiency

Neural network based equalizers can easily compensate channel impairments; such additive noise and inter symbol interference (ISI). The authors present a new approach to improve the training efficiency of the multilayer perceptron (MLP) based equalizer. Their improvement consists on modifying the back propagation (BP) algorithm, by adapting the activation function in addition to the other parameters of the MLP structure. The authors report on experiment results evaluating the performance of the proposed approach namely the back propagation with adaptive activation function (BPAAF) next to the BP algorithm. To further prove its effectiveness, the proposed approach is also compared beside a so known nonlinear equalizer explicitly the multilayer perceptron with decision feedback equalizer MLPDFE. The authors consider various performance measures specifically: signal resorted quality, lower steady state MSE reached and minimum bit error rate (BER) achieved, where nonlinear channel equalization problems are employed.

scholarly journals Comparing Activation Functions in Modeling Shoreline Variation Using Multilayer Perceptron Neural Network

Water ◽  
2020 ◽  
Vol 12 (5) ◽  
pp. 1281
Author(s):  
Je-Chian Chen ◽  
Yu-Min Wang
Keyword(s):  
Neural Network ◽  
Multilayer Perceptron ◽  
Activation Function ◽  
Aerial Survey ◽  
Activation Functions ◽  
Shoreline Changes ◽  
Extensive Evaluation ◽  
Aerial Vehicle ◽  
Sine Functions ◽  
Mlp Model

The study has modeled shoreline changes by using a multilayer perceptron (MLP) neural network with the data collected from five beaches in southern Taiwan. The data included aerial survey maps of the Forestry Bureau for years 1982, 2002, and 2006, which served as predictors, while the unmanned aerial vehicle (UAV) surveyed data of 2019 served as the respondent. The MLP was configured using five different activation functions with the aim of evaluating their significance. These functions were Identity, Tahn, Logistic, Exponential, and Sine Functions. The results have shown that the performance of an MLP model may be affected by the choice of an activation function. Logistic and the Tahn activation functions outperformed the other models, with Logistic performing best in three beaches and Tahn having the rest. These findings suggest that the application of machine learning to shoreline changes should be accompanied by an extensive evaluation of the different activation functions.

scholarly journals FASTER TRAINING USING FUSION OF ACTIVATION FUNCTIONS FOR FEED FORWARD NEURAL NETWORKS

2009 ◽  
Vol 19 (06) ◽  
pp. 437-448 ◽  
Author(s):  
MD. ASADUZZAMAN ◽  
MD. SHAHJAHAN ◽  
KAZUYUKI MURASE
Keyword(s):  
Neural Networks ◽  
Time Series ◽  
Error Function ◽  
Time Series Prediction ◽  
Back Propagation ◽  
Activation Functions ◽  
Classification Problems ◽  
Feed Forward ◽  
Feed Forward Neural Networks ◽  
Prediction Problems

Multilayer feed-forward neural networks are widely used based on minimization of an error function. Back propagation (BP) is a famous training method used in the multilayer networks but it often suffers from the drawback of slow convergence. To make the learning faster, we propose 'Fusion of Activation Functions' (FAF) in which different conventional activation functions (AFs) are combined to compute final activation. This has not been studied extensively yet. One of the sub goals of the paper is to check the role of linear AFs in combination. We investigate whether FAF can enable the learning to be faster. Validity of the proposed method is examined by performing simulations on challenging nine real benchmark classification and time series prediction problems. The FAF has been applied to 2-bit, 3-bit and 4-bit parity, the breast cancer, Diabetes, Heart disease, Iris, wine, Glass and Soybean classification problems. The algorithm is also tested with Mackey-Glass chaotic time series prediction problem. The algorithm is shown to work better than other AFs used independently in BP such as sigmoid (SIG), arctangent (ATAN), logarithmic (LOG).

STUDYING THE EFFECT OF ADAPTIVE MOMENTUM IN IMPROVING THE ACCURACY OF GRADIENT DESCENT BACK PROPAGATION ALGORITHM ON CLASSIFICATION PROBLEMS

2012 ◽  
Vol 09 ◽  
pp. 432-439 ◽  
Author(s):  
MUHAMMAD ZUBAIR REHMAN ◽  
NAZRI MOHD. NAWI
Keyword(s):  
Gradient Descent ◽  
Back Propagation ◽  
Activation Function ◽  
Descent Method ◽  
Learning Rate ◽  
Gradient Descent Method ◽  
Back Propagation Algorithm ◽  
Classification Problems ◽  
Propagation Algorithm ◽  
Adaptive Momentum

Despite being widely used in the practical problems around the world, Gradient Descent Back-propagation algorithm comes with problems like slow convergence and convergence to local minima. Previous researchers have suggested certain modifications to improve the convergence in gradient Descent Back-propagation algorithm such as careful selection of input weights and biases, learning rate, momentum, network topology, activation function and value for 'gain' in the activation function. This research proposed an algorithm for improving the working performance of back-propagation algorithm which is 'Gradient Descent with Adaptive Momentum (GDAM)' by keeping the gain value fixed during all network trials. The performance of GDAM is compared with 'Gradient Descent with fixed Momentum (GDM)' and 'Gradient Descent Method with Adaptive Gain (GDM-AG)'. The learning rate is fixed to 0.4 and maximum epochs are set to 3000 while sigmoid activation function is used for the experimentation. The results show that GDAM is a better approach than previous methods with an accuracy ratio of 1.0 for classification problems like Wine Quality, Mushroom and Thyroid disease.

scholarly journals Prediction of Dengue Hemorrhagic Fever Cases Based on Weather Parameters Using Back Propagation Neural Networks (Case Study in Pontianak City)

2019 ◽  
Vol 15 (2) ◽  
pp. 114-121
Author(s):  
I T Rahayu ◽  
N Nurhasanah ◽  
R Adriat
Keyword(s):  
Neural Networks ◽  
Dengue Hemorrhagic Fever ◽  
Strong Correlation ◽  
Network Architecture ◽  
Back Propagation ◽  
Hemorrhagic Fever ◽  
Activation Function ◽  
Neuron Network ◽  
Back Propagation Neural Networks ◽  
Weather Parameters

Research has been conducted by predicting cases of dengue hemorrhagic fever based on weather parameters. The data used are weather parameters in the form of air temperature data, air humidity, rainfall, duration of solar radiation and wind speed as input data and data on dengue hemorrhagic fever cases as the target data. This study aims to see the confirmation of dengue hemorrhagic parameters in Pontianak. The benefit in the field of education is that students and teachers are aware of the dangers of dengue because it can cause death. The method used is back propagation neural networks with the best network architecture in predicting cases of dengue hemorrhagic fever are [50 40 30 1] and binary sigmoid activation function, bipolar sigmoid and linear function. The activation function will determine whether the signal from the neuron input will be forwarded to other neurons and is also used to determine the output of a neuron. Network training correlation value is 0.9995 (very strong correlation) with MSE 0.0001 and network testing is 0.9325 (very strong correlation) with MSE 1.61. Determination coefficient serve as accuracy with values obtained is 0.85, which means that 85% of weather parameters can be used as input in predicting the incidence of dengue hemorrhagic fever in Pontianak City.

A New Modified Back-Propagation Algorithm for Forecasting Malaysian Housing Demand

2012 ◽  
Vol 232 ◽  
pp. 908-912 ◽  
Author(s):  
Nazri Mohd Nawi ◽  
Noorhamreeza Abdul Hamid ◽  
Noor Yasmin Zainun
Keyword(s):  
Traditional Approach ◽  
Low Cost ◽  
Back Propagation ◽  
Activation Function ◽  
Peninsular Malaysia ◽  
Housing Demand ◽  
Back Propagation Algorithm ◽  
New Approach ◽  
Gradient Based ◽  
Low Cost Housing

Over the past decade, the growth of the housing construction in Malaysia has been increase dramatically and the level of urbanization process in Malaysia is considered to be important in planning for low-cost housing needs. Unfortunately, there is a clear miss-match between the supply and the demand of low cost housing in Malaysia. Due to the problems faced, there have been several attempts in predicting housing demands using the artificial-neural networks (ANN) technique particularly back-propagation (BP). However, the training process of BP can result in slow convergence or even network paralysis and can easily get stuck at local minima. This paper presents a new approach to improve the training efficiency of BP algorithms to forecast low-cost housing demand in one of the states in Peninsular Malaysia. The proposed algorithm (BPM/AG) adaptively modifies the gradient based search direction by introducing the value of gain parameter in the activation function. The results show that the proposed algorithm significantly improves the learning process with more than 31% faster in term of CPU time and number of epochs as compared to the traditional approach. The proposed algorithm can forecast low-cost housing demand very well with 6.62% of MAPE value.

scholarly journals Broad Autoencoder Features Learning for Classification Problem

2021 ◽  
Vol 15 (4) ◽  
pp. 0-0
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Classification Problem ◽  
Activation Function ◽  
Activation Functions ◽  
Classification Problems ◽  
Stacked Autoencoders ◽  
Learned Features ◽  
Sigmoid Functions ◽  
Nonlinear Mappings

Activation functions such as Tanh and Sigmoid functions are widely used in Deep Neural Networks (DNNs) and pattern classification problems. To take advantages of different activation functions, the Broad Autoencoder Features (BAF) is proposed in this work. The BAF consists of four parallel-connected Stacked Autoencoders (SAEs) and each of them uses a different activation function, including Sigmoid, Tanh, ReLU, and Softplus. The final learned features can merge such features by various nonlinear mappings from original input features with such a broad setting. This helps to excavate more information from the original input features. Experimental results show that the BAF yields better-learned features and classification performances.

scholarly journals Jaringan Saraf Tiruan dengan Backpropagation untuk Memprediksi Penyakit Asma

2015 ◽  
Vol 2 (1) ◽  
pp. 28
Author(s):  
Dahriani Hakim Tanjung
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Network Architecture ◽  
Back Propagation ◽  
Activation Function ◽  
Learning Rate ◽  
Sigmoid Function ◽  
A Value ◽  
Artificial Neural ◽  
Require Output

Penelitian ini bertujuan untuk memprediksi penyakit asma menggunakan teknik pengenalan pola yaitu jaringan saraf tiruan dengan metode backpropagation. Data penilaian asma mengacu pada riwayat penyakit asma seseorang. Jaringan saraf tiruan dilakukan dengan menentukan jumlah unit untuk setiap lapisan dengan fungsi aktivasi sigmoid biner. Pengujian dilakukan menggunakan perangkat lunak matlab yang diuji dengan beberapa bentuk arsitektur jaringan. Arsitektur dengan konfigurasi terbaik terdiri dari 18 lapisan masukan, 8 lapisan tersembunyi dan 4 lapisan keluaran dengan nilai learning rate sebesar 0.5, nilai toleransi error 0.001, menghasilkan maksimal epoch 4707 dan MSE 0.00100139. MSE berada di bawah nilai error yaitu 0.001, Parameter tersebut dipilih menjadi parameter terbaik karena menghasilkan jumlah iterasi yang memiliki nilai akurasi MSE yang cukup baik, karena nilai MSE paling kecil dari arsitektur yang lain serta nilai MSE dibawah dari nilai error yang ditentukan. Sigmoid Biner Fungsi ini digunakan untuk jaringan saraf yang dilatih dengan menggunakan metode backpropagation. Fungsi sigmoid memiliki nilai range 0 sampai 1. Oleh karena itu, fungsi ini sering digunakan untuk jaringan saraf yang membutuhkan nilai output yang terletak pada interval 0 sampai 1.This study aims to predict asthma using pattern recognition techniques namely artificial neural network with back propagation method. Asthma assessment data refers to a person's history of asthma. Artificial neural network is done by determining the number of units for each layer with binary sigmoid activation function. Testing is done using matlab software being tested with some form of network architecture. Architecture with the best configuration consists of 18 layers of input, 8 hidden layer and output layer 4 with a value of learning rate of 0.5, the error tolerance value 0001, 4707 and resulted in the maximum epoch MSE .00100139. MSE is under the error value is 0.001, the parameter is chosen to be the best parameters for generating the number of iterations that have an accuracy value of MSE is quite good, because the smallest MSE value than other architectures as well as the value of the MSE under a specified error value. Binary sigmoid function is used for neural network trained using the backpropagation method. Sigmoid function has a value in the range 0 to 1. Therefore, this function is often used for neural networks that require output value lies in the interval 0 to 1.

Sign in / Sign up

Export Citation Format

Share Document