scholarly journals Ensemble of convolutional neural networks trained with different activation functions

2021 ◽  
Vol 166 ◽  
pp. 114048
Author(s):  
Gianluca Maguolo ◽  
Loris Nanni ◽  
Stefano Ghidoni
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Activation Functions

scholarly journals Envision Foundational of Convolution Neural Network

2021 ◽  
Vol 10 (6) ◽  
pp. 54-60
Author(s):  
M Venkata Krishna Reddy* ◽  
Pradeep S.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Computer Graphics ◽  
Convolutional Neural Networks ◽  
Generative Models ◽  
Activation Functions ◽  
Machine Learning Classification ◽  
Network Intrusion ◽  
Recent Advances

1. Bilal, A. Jourabloo, M. Ye, X. Liu, and L. Ren. Do Convolutional Neural Networks Learn Class Hierarchy? IEEE Transactions on Visualization and Computer Graphics, 24(1):152–162, Jan. 2018. 2. M. Carney, B. Webster, I. Alvarado, K. Phillips, N. Howell, J. Griffith, J. Jongejan, A. Pitaru, and A. Chen. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20. ACM, Honolulu, HI, USA, 2020. 3. A. Karpathy. CS231n Convolutional Neural Networks for Visual Recognition, 2016 4. M. Kahng, N. Thorat, D. H. Chau, F. B. Viegas, and M. Wattenberg. GANLab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation. IEEE Transactions on Visualization and Computer Graphics, 25(1):310–320, Jan. 2019. 5. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015 6. M. Kahng, P. Y. Andrews, A. Kalro, and D. H. Chau. ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. IEEE Transactions on Visualization and Computer Graphics, 24(1):88–97, Jan. 2018. 7. https://cs231n.github.io/convolutional-networks/ 8. https://www.analyticsvidhya.com/blog/2020/02/learn-imageclassification-cnn-convolutional-neural-networks-3-datasets/ 9. https://towardsdatascience.com/understanding-cnn-convolutionalneural- network-69fd626ee7d4 10. https://medium.com/@birdortyedi_23820/deep-learning-lab-episode-2- cifar- 10-631aea84f11e 11. J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen. Recent advances in convolutional neural networks. Pattern Recognition, 77:354–377, May 2018. 12. Hamid, Y., Shah, F.A. and Sugumaram, M. (2014), ―Wavelet neural network model for network intrusion detection system‖, International Journal of Information Technology, Vol. 11 No. 2, pp. 251-263 13. G Sreeram , S Pradeep, K SrinivasRao , B.Deevan Raju , Parveen Nikhat , ― Moving ridge neuronal espionage network simulation for reticulum invasion sensing‖. International Journal of Pervasive Computing and Communications.https://doi.org/10.1108/IJPCC-05- 2020-0036 14. E. Stevens, L. Antiga, and T. Viehmann. Deep Learning with PyTorch. O’Reilly Media, 2019. 15. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015. 16. Aman Dureja, Payal Pahwa, ―Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks‖, Recent Advances in Computer Science , Vol 2, Issue 3, 2019 ,PP-156-161 17. https://missinglink.ai/guides/neural-network-concepts/7-types-neuralnetwork-activation-functions-right/

Adaptive activation functions in convolutional neural networks

2018 ◽  
Vol 272 ◽  
pp. 204-212 ◽  
Author(s):  
Sheng Qian ◽  
Hua Liu ◽  
Cheng Liu ◽  
Si Wu ◽  
Hau San Wong
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Activation Functions

scholarly journals An Adiabatic Method to Train Binarized Artificial Neural Networks

2021 ◽  
Author(s):  
Jiang Xiao ◽  
Yuansheng Zhao
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Training Algorithms ◽  
Training Method ◽  
Dense Network ◽  
Activation Functions ◽  
Hyperbolic Tangent ◽  
Different Types ◽  
Artificial Neural ◽  
Hand Writing

Abstract An artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (reLU) functions, etc. Synapses connect the neuron outputs to their inputs with tunable real-valued weights. The most resource-demanding operations in realizing such neural networks are the multiplication and accumulate (MAC) operations that compute the dot product be- tween real-valued outputs from neurons and the synapses weights. The efficiency of neural networks can be drastically enhanced if the neuron outputs and/or the weights can be trained to take binary values ±1 only, for which the MAC can be replaced by the simple XOR operations. In this paper, we demonstrate an adiabatic training method that can successfully binarize the dense neural networks and the convolutional neural networks without modification in terms network structure and with very minimal change in training algorithms. This adiabatic training method is tested in the following four tasks: the recognition of hand-writing numbers using a usual dense network, the cat-dog recog- nition and the audio recognition using a convolutional neural networks, the image recognition with 10 classes (CIFAR-10) using ResNet20 and VGG-Small networks. In all tasks, the performance of the binary neural networks trained by the adiabatic method are almost identical to the networks trained using the conventional reLU or Sigmoid activations with real-valued activations and weights. This adiabatic method can be easily applied to binarize different types of networks, and will increase the computational efficiency considerably and greatly simplify the deployment of neural networks.

scholarly journals An adiabatic method to train binarized artificial neural networks

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Yuansheng Zhao ◽  
Jiang Xiao
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Training Algorithms ◽  
Training Method ◽  
Activation Functions ◽  
Hyperbolic Tangent ◽  
Different Types ◽  
Artificial Neural ◽  
Fully Connected ◽  
Hand Writing

AbstractAn artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (ReLU) functions, etc.. Synapses connect the neuron outputs to their inputs with tunable real-valued weights. The most resource-demanding operations in realizing such neural networks are the multiplication and accumulate (MAC) operations that compute the dot product between real-valued outputs from neurons and the synapses weights. The efficiency of neural networks can be drastically enhanced if the neuron outputs and/or the weights can be trained to take binary values $$\pm 1$$ ± 1 only, for which the MAC can be replaced by the simple XNOR operations. In this paper, we demonstrate an adiabatic training method that can binarize the fully-connected neural networks and the convolutional neural networks without modifying the network structure and size. This adiabatic training method only requires very minimal changes in training algorithms, and is tested in the following four tasks: the recognition of hand-writing numbers using a usual fully-connected network, the cat-dog recognition and the audio recognition using convolutional neural networks, the image recognition with 10 classes (CIFAR-10) using ResNet-20 and VGG-Small networks. In all tasks, the performance of the binary neural networks trained by the adiabatic method are almost identical to the networks trained using the conventional ReLU or Sigmoid activations with real-valued activations and weights. This adiabatic method can be easily applied to binarize different types of networks, and will increase the computational efficiency considerably and greatly simplify the deployment of neural networks.

scholarly journals Improving Convolutional Neural Networks with Competitive Activation Function

2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Yao Ying ◽  
Nengbo Zhang ◽  
Ping He ◽  
Silong Peng
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Activation Function ◽  
Linear Mapping ◽  
Nonlinear Transformation ◽  
Nonlinear Mapping ◽  
Activation Functions ◽  
Competition Mechanism ◽  
Function Design ◽  
Competitive Activation

The activation function is the basic component of the convolutional neural network (CNN), which provides the nonlinear transformation capability required by the network. Many activation functions make the original input compete with different linear or nonlinear mapping terms to obtain different nonlinear transformation capabilities. Until recently, the original input of funnel activation (FReLU) competed with the spatial conditions, so FReLU not only has the ability of nonlinear transformation but also has the ability of pixelwise modeling. We summarize the competition mechanism in the activation function and then propose a novel activation function design template: competitive activation function (CAF), which promotes competition among different elements. CAF generalizes all activation functions that use competition mechanisms. According to CAF, we propose a parametric funnel rectified exponential unit (PFREU). PFREU promotes competition among linear mapping, nonlinear mapping, and spatial conditions. We conduct experiments on four datasets of different sizes, and the experimental results of three classical convolutional neural networks proved the superiority of our method.

Sign in / Sign up

Export Citation Format

Share Document