Neural Network Activation Functions with Electro-Optic Absorption Modulators

Activation functions are fundamental elements in artificial neural networks. The mathematical formulation of some activation functions (e.g. Heaviside function and Rectified Linear Unit function) are not expressed in an explicit closed form. This made them numerically unstable and computationally complex during estimation. This paper introduces a novel explicit analytic form for those activation functions. The proposed mathematical equations match exactly the original definition of the studied activation function. The proposed equations can be adapted better in optimization, forward and backward propagation algorithm employed in an artificial neural network.

Download Full-text

Artificial Neural Network Activation Functions in Exact Analytical Form (Heaviside, ReLU, PReLU, ELU, SELU, ELiSH)

10.36227/techrxiv.15096888 ◽

2021 ◽

Author(s):

Rami Alkhatib

Keyword(s):

Neural Network ◽

Artificial Neural Network ◽

Mathematical Formulation ◽

Analytic Form ◽

Analytical Form ◽

Heaviside Function ◽

Activation Functions ◽

Network Activation ◽

Original Definition ◽

Artificial Neural

Activation functions are fundamental elements in artificial neural networks. The mathematical formulation of some activation functions (e.g. Heaviside function and Rectified Linear Unit function) are not expressed in an explicit closed form. This made them numerically unstable and computationally complex during estimation. This paper introduces a novel explicit analytic form for those activation functions. The proposed mathematical equations match exactly the original definition of the studied activation function. The proposed equations can be adapted better in optimization, forward and backward propagation algorithm employed in an artificial neural network.

Download Full-text

Comparison of Different Convolutional Neural Network Activation Functions and Methods for Building Ensembles

10.20944/preprints202103.0180.v1 ◽

2021 ◽

Author(s):

Loris Nanni ◽

Sheryl Brahnam ◽

Michelangelo Paci ◽

Gianluca Maguolo

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Comprehensive Evaluation ◽

Source Code ◽

Data Sets ◽

Biomedical Data ◽

Activation Functions ◽

Highly Efficient ◽

Network Activation ◽

Classification Tasks

Recently, much attention has been devoted to finding highly efficient and powerful activation functions for CNN layers. Because activation functions inject different nonlinearities between layers that affect performance, varying them is one method for building robust ensembles of CNNs. The objective of this study is to examine the performance of CNN ensembles made with different activation functions, including six new ones presented here: 2D Mexican ReLU, TanELU, MeLU+GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The highest performing ensemble was built with CNNs having different activation layers that randomly replaced the standard ReLU. A comprehensive evaluation of the proposed approach was conducted across fifteen biomedical data sets representing various classification tasks. The proposed method was tested on two basic CNN architectures: Vgg16 and ResNet50. Results demonstrate the superiority in performance of this approach. The MATLAB source code for this study will be available at https://github.com/LorisNanni.

Download Full-text