A Survey on Efficient Realization of Activation Functions of Artificial Neural Network

2021 ◽  
Author(s):  
Raghuvendra Pratap Tripathi ◽  
Manish Tiwari ◽  
Amit Dhawan ◽  
Anand Sharma ◽  
Sumit Kumar Jha
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Activation Functions ◽  
Artificial Neural ◽  
Efficient Realization

scholarly journals One-time learning in a biologically-inspired Salience-affected Artificial Neural Network (SANN)

2019 ◽  
Author(s):  
Leendert A Remmelzwaal ◽  
George F R Ellis ◽  
Jonathan Tapson
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Neural Dynamics ◽  
Proof Of Concept ◽  
Activation Functions ◽  
Learning Networks ◽  
Biologically Inspired ◽  
Artificial Neural ◽  
Validation Testing ◽  
And Robotics

AbstractIn this paper we introduce a novel Salience Affected Artificial Neural Network (SANN) that models the way neuromodulators such as dopamine and noradrenaline affect neural dynamics in the human brain by being distributed diffusely through neocortical regions. This allows one-time learning to take place through strengthening entire patterns of activation at one go. We present a model that accepts a salience signal, and returns a reverse salience signal. We demonstrate that we can tag an image with salience with only a single training iteration, and that the same image will then produces the highest reverse salience signal during classification. We explore the effects of salience on learning via its effect on the activation functions of each node, as well as on the strength of weights in the network. We demonstrate that a salience signal improves classification accuracy of the specific image that was tagged with salience, as well as all images in the same class, while penalizing images in other classes. Results are validated using 5-fold validation testing on MNIST and Fashion MNIST datasets. This research serves as a proof of concept, and could be the first step towards introducing salience tagging into Deep Learning Networks and robotics.

scholarly journals An Artificial Neural Network Model to Predict the Thermal Properties of Concrete Using Different Neurons and Activation Functions

2019 ◽  
Vol 2019 ◽  
pp. 1-13 ◽  
Author(s):  
Sehmus Fidan ◽  
Hasan Oktay ◽  
Suleyman Polat ◽  
Sarper Ozturk
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Thermal Properties ◽  
Thermal Property ◽  
Building Materials ◽  
Performance Indicator ◽  
Mechanical And Thermal Properties ◽  
Activation Functions ◽  
Marquardt Algorithm ◽  
Artificial Neural

Growing concerns on energy consumption of buildings by heating and cooling applications have led to a demand for improved insulating performances of building materials. The establishment of thermal property for a building structure is the key performance indicator for energy efficiency, whereas high accuracy and precision tests are required for its determination which increases time and experimental costs. The main scope of this study is to develop a model based on artificial neural network (ANN) in order to predict the thermal properties of concrete through its mechanical characteristics. Initially, different concrete samples were prepared, and their both mechanical and thermal properties were tested in accordance with ASTM and EN standards. Then, the Levenberg–Marquardt algorithm was used for training the neural network in the single hidden layer using 5, 10, 15, 20, and 25 neurons, respectively. For each thermal property, various activation functions such as tangent sigmoid functions and triangular basis functions were used to examine the best solution performance. Moreover, a cross-validation technique was used to ensure good generalization and to avoid overtraining. ANN results showed that the best overall R2 performances for the prediction of thermal conductivity, specific heat, and thermal diffusivity were obtained as 0.996, 0.983, and 0.995 for tansig activation functions with 25, 25, and 20 neurons, respectively. The performance results showed that there was a great consistency between the predicted and tested results, demonstrating the feasibility and practicability of the proposed ANN models for predicting the thermal property of a concrete.

scholarly journals Advanced Stochastic Optimization Algorithm for Deep Learning Artificial Neural Networks in Banking and Finance Industries

2019 ◽  
Vol 1 (1) ◽  
pp. p8
Author(s):  
Jamilu Auwalu Adamu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Network ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Probability Distribution ◽  
Trial And Error ◽  
Activation Functions ◽  
Data Set ◽  
Artificial Neural

One of the objectives of this paper is to incorporate fat-tail effects into, for instance, Sigmoid in order to introduce Transparency and Stability into the existing stochastic Activation Functions. Secondly, according to the available literature reviewed, the existing set of Activation Functions were introduced into the Deep learning Artificial Neural Network through the “Window” not properly through the “Legitimate Door” since they are “Trial and Error “and “Arbitrary Assumptions”, thus, the Author proposed a “Scientific Facts”, “Definite Rules: Jameel’s Stochastic ANNAF Criterion”, and a “Lemma” to substitute not necessarily replace the existing set of stochastic Activation Functions, for instance, the Sigmoid among others. This research is expected to open the “Black-Box” of Deep Learning Artificial Neural networks. The author proposed a new set of advanced optimized fat-tailed Stochastic Activation Functions EMANATED from the AI-ML-Purified Stocks Data  namely; the Log – Logistic (3P) Probability Distribution (1st), Cauchy Probability Distribution (2nd), Pearson 5 (3P) Probability Distribution (3rd), Burr (4P) Probability Distribution (4th), Fatigue Life (3P) Probability Distribution (5th), Inv. Gaussian (3P) Probability Distribution (6th), Dagum (4P) Probability Distribution (7th), and Lognormal (3P) Probability Distribution (8th) for the successful conduct of both Forward and Backward Propagations of Deep Learning Artificial Neural Network. However, this paper did not check the Monotone Differentiability of the proposed distributions. Appendix A, B, and C presented and tested the performances of the stressed Sigmoid and the Optimized Activation Functions using Stocks Data (2014-1991) of Microsoft Corporation (MSFT), Exxon Mobil (XOM), Chevron Corporation (CVX), Honda Motor Corporation (HMC), General Electric (GE), and U.S. Fundamental Macroeconomic Parameters, the results were found fascinating. Thus, guarantee, the first three distributions are excellent Activation Functions to successfully conduct any Stock Deep Learning Artificial Neural Network. Distributions Number 4 to 8 are also good Advanced Optimized Activation Functions. Generally, this research revealed that the Advanced Optimized Activation Functions satisfied Jameel’s ANNAF Stochastic Criterion depends on the Referenced Purified AI Data Set, Time Change and Area of Application which is against the existing “Trial and Error “and “Arbitrary Assumptions” of Sigmoid, Tanh, Softmax, ReLu, and Leaky ReLu.

Definition of Artificial Neural Network

2018 ◽  
pp. 32-46
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Artificial Neural Network ◽  
Learning Tools ◽  
Activation Functions ◽  
Mathematical Definition ◽  
Artificial Neural ◽  
Artificial Neural Network Ann ◽  
Definition Of

This chapter is an explanation of artificial neural network (ANN), which is one of the machine learning tools applied for medical purposes. The biological and mathematical definition of neural network is provided and the activation functions effective for processing are listed. Some figures are collected for better understanding.

scholarly journals Deep Convolutional Neural Network Algorithm for Prediction of the Mechanical Properties of Friction Stir Welded Copper Joints from its Microstructures

2021 ◽  
Author(s):  
Akshansh Mishra ◽  
Asmita Suman
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Convolutional Neural Network ◽  
Semantic Segmentation ◽  
Validation Dataset ◽  
Activation Functions ◽  
Friction Stir ◽  
Neural Network Algorithm ◽  
Artificial Neural ◽  
Detection And Localization

Abstract Convolutional Neural Network (CNN) is a special type of Artificial Neural Network which takes input in the form of an image. Like Artificial Neural Network they consist of weights that are estimated during training, neurons (activation functions), and an objective (loss function). CNN is finding various applications in image recognition, semantic segmentation, object detection, and localization. The present work deals with the prediction of the welding efficiency of the Friction Stir Welded joints on the basis of microstructure images by carrying out training on 3000 microstructure images and further testing on 300 microstructure images. The obtained results showed an accuracy of 80 % on the validation dataset.

scholarly journals Artificial Neural Network Activation Functions in Exact Analytical Form (Heaviside, ReLU, PReLU, ELU, SELU, ELiSH)

2021 ◽  
Author(s):  
Rami Alkhatib
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Mathematical Formulation ◽  
Analytic Form ◽  
Analytical Form ◽  
Heaviside Function ◽  
Activation Functions ◽  
Network Activation ◽  
Original Definition ◽  
Artificial Neural

Activation functions are fundamental elements in artificial neural networks. The mathematical formulation of some activation functions (e.g. Heaviside function and Rectified Linear Unit function) are not expressed in an explicit closed form. This made them numerically unstable and computationally complex during estimation. This paper introduces a novel explicit analytic form for those activation functions. The proposed mathematical equations match exactly the original definition of the studied activation function. The proposed equations can be adapted better in optimization, forward and backward propagation algorithm employed in an artificial neural network.

Sign in / Sign up

Export Citation Format

Share Document