scholarly journals Phish: A Novel Hyper-Optimizable Activation Function

2021 ◽  
Author(s):  
Philip Naveen
Keyword(s):  
Large Scale ◽  
Deep Neural Networks ◽  
Law Of Large Numbers ◽  
Statistical Tests ◽  
Activation Function ◽  
Activation Functions ◽  
Composite Function ◽  
The Past ◽  
Large Numbers ◽  
Generalized Networks

Deep-learning models estimate values using backpropagation. The activation function within hidden layers is a critical component to minimizing loss in deep neural-networks. Rectified Linear (ReLU) has been the dominant activation function for the past decade. Swish and Mish are newer activation functions that have shown to yield better results than ReLU given specific circumstances. Phish is a novel activation function proposed here. It is a composite function defined as f(x) = xTanH(GELU(x)), where no discontinuities are apparent in the differentiated graph on the domain observed. Four generalized networks were constructed using Phish, Swish, Sigmoid, and TanH. SoftMax was the output function. Using images from MNIST and CIFAR-10 databanks, these networks were trained to minimize sparse categorical crossentropy. A large scale cross-validation was simulated using stochastic Markov chains to account for the law of large numbers for the probability values. Statistical tests support the research hypothesis stating Phish could outperform other activation functions in classification. Future experiments would involve testing Phish in unsupervised learning algorithms and comparing it to more activation functions.

scholarly journals Phish: A Novel Hyper-Optimizable Activation Function

2021 ◽  
Author(s):  
Philip Naveen
Keyword(s):  
Large Scale ◽  
Deep Neural Networks ◽  
Law Of Large Numbers ◽  
Statistical Tests ◽  
Activation Function ◽  
Activation Functions ◽  
Composite Function ◽  
The Past ◽  
Large Numbers ◽  
Generalized Networks

Deep-learning models estimate values using backpropagation. The activation function within hidden layers is a critical component to minimizing loss in deep neural-networks. Rectified Linear (ReLU) has been the dominant activation function for the past decade. Swish and Mish are newer activation functions that have shown to yield better results than ReLU given specific circumstances. Phish is a novel activation function proposed here. It is a composite function defined as f(x) = xTanH(GELU(x)), where no discontinuities are apparent in the differentiated graph on the domain observed. Four generalized networks were constructed using Phish, Swish, Sigmoid, and TanH. SoftMax was the output function. Using images from MNIST and CIFAR-10 databanks, these networks were trained to minimize sparse categorical crossentropy. A large scale cross-validation was simulated using stochastic Markov chains to account for the law of large numbers for the probability values. Statistical tests support the research hypothesis stating Phish could outperform other activation functions in classification. Future experiments would involve testing Phish in unsupervised learning algorithms and comparing it to more activation functions.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

scholarly journals Trigonometric Inference Providing Learning in Deep Neural Networks

2021 ◽  
Vol 11 (15) ◽  
pp. 6704
Author(s):  
Jingyong Cai ◽  
Masashi Takemoto ◽  
Yuming Qiu ◽  
Hironori Nakajo
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Trigonometric Approximation ◽  
Model Parameters ◽  
Training Algorithms ◽  
Activation Functions ◽  
Classical Training ◽  
Sum Formula

Despite being heavily used in the training of deep neural networks (DNNs), multipliers are resource-intensive and insufficient in many different scenarios. Previous discoveries have revealed the superiority when activation functions, such as the sigmoid, are calculated by shift-and-add operations, although they fail to remove multiplications in training altogether. In this paper, we propose an innovative approach that can convert all multiplications in the forward and backward inferences of DNNs into shift-and-add operations. Because the model parameters and backpropagated errors of a large DNN model are typically clustered around zero, these values can be approximated by their sine values. Multiplications between the weights and error signals are transferred to multiplications of their sine values, which are replaceable with simpler operations with the help of the product to sum formula. In addition, a rectified sine activation function is utilized for further converting layer inputs into sine values. In this way, the original multiplication-intensive operations can be computed through simple add-and-shift operations. This trigonometric approximation method provides an efficient training and inference alternative for devices with insufficient hardware multipliers. Experimental results demonstrate that this method is able to obtain a performance close to that of classical training algorithms. The approach we propose sheds new light on future hardware customization research for machine learning.

Pulsar Detection for Wavelets SODA and Regularized Fuzzy Neural Networks Based on Andneuron and Robust Activation Function

2019 ◽  
Vol 28 (01) ◽  
pp. 1950003 ◽  
Author(s):  
Paulo Vitor de Campos Souza ◽  
Luiz Carlos Bambirra Torres ◽  
Augusto Junio Guimarães ◽  
Vanessa Souza Araujo
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
Activation Function ◽  
Fuzzy Neural Networks ◽  
Activation Functions ◽  
Large Scale Problems ◽  
Proposed Model ◽  
Fuzzy Neural ◽  
Learning Machine ◽  
Intelligent Models

The use of intelligent models may be slow because of the number of samples involved in the problem. The identification of pulsars (stars that emit Earth-catchable signals) involves collecting thousands of signals by professionals of astronomy and their identification may be hampered by the nature of the problem, which requires many dimensions and samples to be analyzed. This paper proposes the use of hybrid models based on concepts of regularized fuzzy neural networks that use the representativeness of input data to define the groupings that make up the neurons of the initial layers of the model. The andneurons are used to aggregate the neurons of the first layer and can create fuzzy rules. The training uses fast extreme learning machine concepts to generate the weights of neurons that use robust activation functions to perform pattern classification. To solve large-scale problems involving the nature of pulsar detection problems, the model proposes a fast and highly accurate approach to address complex issues. In the execution of the tests with the proposed model, experiments were conducted explanation in two databases of pulsars, and the results prove the viability of the fast and interpretable approach in identifying such involved stars.

scholarly journals Rescuing Deep Hashing from Dead Bits Problem

2021 ◽  
Author(s):  
Shu Zhao ◽  
Dayan Wu ◽  
Yucan Zhou ◽  
Bo Li ◽  
Weiping Wang
Keyword(s):  
Large Scale ◽  
Activation Function ◽  
Activation Functions ◽  
Retrieval Accuracy ◽  
Deep Hashing ◽  
Common Strategy ◽  
Negative Effect ◽  
Gradient Amplifier ◽  
Discrete Values ◽  
Large Scale Image Retrieval

Deep hashing methods have shown great retrieval accuracy and efficiency in large-scale image retrieval. How to optimize discrete hash bits is always the focus in deep hashing methods. A common strategy in these methods is to adopt an activation function, e.g. sigmoid() or tanh(), and minimize a quantization loss to approximate discrete values. However, this paradigm may make more and more hash bits stuck into the wrong saturated area of the activation functions and never escaped. We call this problem "Dead Bits Problem (DBP)". Besides, the existing quantization loss will aggravate DBP as well. In this paper, we propose a simple but effective gradient amplifier which acts before activation functions to alleviate DBP. Moreover, we devise an error-aware quantization loss to further alleviate DBP. It avoids the negative effect of quantization loss based on the similarity between two images. The proposed gradient amplifier and error-aware quantization loss are compatible with a variety of deep hashing methods. Experimental results on three datasets demonstrate the efficiency of the proposed gradient amplifier and the error-aware quantization loss.

scholarly journals Differential Equation Units: Learning Functional Forms of Activation Functions from Data

2020 ◽  
Vol 34 (04) ◽  
pp. 6030-6037
Author(s):  
MohamadAli Torkamani ◽  
Shiv Shankar ◽  
Amirmohammad Rooshenas ◽  
Phillip Wallis
Keyword(s):  
Differential Equation ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Functional Form ◽  
Activation Function ◽  
The Other ◽  
Superior Performance ◽  
Activation Functions ◽  
Functional Forms ◽  
Nonlinear Activation Function

Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation function from a family of solutions to an ordinary differential equation. Specifically, each neuron may change its functional form during training based on the behavior of the other parts of the network. We show that using neurons with DEU activation functions results in a more compact network capable of achieving comparable, if not superior, performance when compared to much larger networks.

scholarly journals Overview of Configuring Adaptive Activation Functions for Deep Neural Networks - A Comparative Study

2021 ◽  
Vol 3 (1) ◽  
pp. 10-22
Author(s):  
Wang Haoxiang ◽  
Smys S
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Learning Process ◽  
Deep Neural Networks ◽  
Research Work ◽  
Nonlinear Function ◽  
Activation Function ◽  
Classification Error ◽  
Activation Functions ◽  
Research Article

Recently, the deep neural networks (DNN) have demonstrated many performances in the pattern recognition paradigm. The research studies on DNN include depth layer networks, filters, training and testing datasets. Deep neural network is providing many solutions for nonlinear partial differential equations (PDE). This research article comprises of many activation functions for each neuron. Besides, these activation networks are allowing many neurons within the neuron networks. In this network, the multitude of the functions will be selected between node by node to minimize the classification error. This is the reason for selecting the adaptive activation function for deep neural networks. Therefore, the activation functions are adapted with every neuron on the network, which is used to reduce the classification error during the process. This research article discusses the scaling factor for activation function that provides better optimization for the process in the dynamic changes of procedure. The proposed adaptive activation function has better learning capability than fixed activation function in any neural network. The research articles compare the convergence rate, early training function, and accuracy between existing methods. Besides, this research work provides improvements in debt ideas of the learning process of various neural networks. This learning process works and tests the solution available in the domain of various frequency bands. In addition to that, both forward and inverse problems of the parameters in the overriding equation will be identified. The proposed method is very simple architecture and efficiency, robustness, and accuracy will be high when considering the nonlinear function. The overall classification performance will be improved in the resulting networks, which have been trained with common datasets. The proposed work is compared with the recent findings in neuroscience research and proved better performance.

scholarly journals Broad Autoencoder Features Learning for Classification Problem

2021 ◽  
Vol 15 (4) ◽  
pp. 0-0
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Classification Problem ◽  
Activation Function ◽  
Activation Functions ◽  
Classification Problems ◽  
Stacked Autoencoders ◽  
Learned Features ◽  
Sigmoid Functions ◽  
Nonlinear Mappings

Activation functions such as Tanh and Sigmoid functions are widely used in Deep Neural Networks (DNNs) and pattern classification problems. To take advantages of different activation functions, the Broad Autoencoder Features (BAF) is proposed in this work. The BAF consists of four parallel-connected Stacked Autoencoders (SAEs) and each of them uses a different activation function, including Sigmoid, Tanh, ReLU, and Softplus. The final learned features can merge such features by various nonlinear mappings from original input features with such a broad setting. This helps to excavate more information from the original input features. Experimental results show that the BAF yields better-learned features and classification performances.

scholarly journals Stochastic Selection of Activation Layers for Convolutional Neural Networks

Sensors ◽  
2020 ◽  
Vol 20 (6) ◽  
pp. 1626 ◽  
Author(s):  
Loris Nanni ◽  
Alessandra Lumini ◽  
Stefano Ghidoni ◽  
Gianluca Maguolo
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Computational Time ◽  
Activation Functions ◽  
Practical Applications ◽  
Functional Blocks ◽  
Stochastic Selection ◽  
Selection Of

In recent years, the field of deep learning has achieved considerable success in pattern recognition, image segmentation, and many other classification fields. There are many studies and practical applications of deep learning on images, video, or text classification. Activation functions play a crucial role in discriminative capabilities of the deep neural networks and the design of new “static” or “dynamic” activation functions is an active area of research. The main difference between “static” and “dynamic” functions is that the first class of activations considers all the neurons and layers as identical, while the second class learns parameters of the activation function independently for each layer or even each neuron. Although the “dynamic” activation functions perform better in some applications, the increased number of trainable parameters requires more computational time and can lead to overfitting. In this work, we propose a mixture of “static” and “dynamic” activation functions, which are stochastically selected at each layer. Our idea for model design is based on a method for changing some layers along the lines of different functional blocks of the best performing CNN models, with the aim of designing new models to be used as stand-alone networks or as a component of an ensemble. We propose to replace each activation layer of a CNN (usually a ReLU layer) by a different activation function stochastically drawn from a set of activation functions: in this way, the resulting CNN has a different set of activation function layers.

Sign in / Sign up

Export Citation Format

Share Document