Background:
In making the deep neural network, activation functions play an important
role. But the choice of activation functions also affects the network in term of optimization and to
retrieve the better results. Several activation functions have been introduced in machine learning for
many practical applications. But which activation function should use at hidden layer of deep neural
networks was not identified.
Objective:
The primary objective of this analysis was to describe which activation function must be
used at hidden layers for deep neural networks to solve complex non-linear problems.
Methods:
The configuration for this comparative model was used by using the datasets of 2 classes
(Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was
also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts.
The first 8000 images were mainly used for training the network and the next 2000 images were
used for testing the network.
Results:
The experimental comparison was done by analyzing the network by taking different activation
functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset
were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden
layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation
accuracy at 25th Epoch 0.8320.
Conclusion:
It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives
best results and improve overall performance better in term of accuracy and speed. These advantages
of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images
from the databases.
Non‐linear activation function approximation using a REMEZ algorithm
