Analysis of Nonlinear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
pp. 1179-1190
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Activation Functions ◽  
Classification Tasks

scholarly journals Inter-Class Angular Loss for Convolutional Neural Networks

2019 ◽  
Vol 33 ◽  
pp. 3894-3901 ◽  
Author(s):  
Le Hui ◽  
Xiang Li ◽  
Chen Gong ◽  
Meng Fang ◽  
Joey Tianyi Zhou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Deep Neural Networks ◽  
Learning Difficulties ◽  
Feature Space ◽  
Superior Performance ◽  
Strongly Correlated ◽  
Discriminative Ability ◽  
Practical Applications ◽  
Classification Tasks

Convolutional Neural Networks (CNNs) have shown great power in various classification tasks and have achieved remarkable results in practical applications. However, the distinct learning difficulties in discriminating different pairs of classes are largely ignored by the existing networks. For instance, in CIFAR-10 dataset, distinguishing cats from dogs is usually harder than distinguishing horses from ships. By carefully studying the behavior of CNN models in the training process, we observe that the confusion level of two classes is strongly correlated with their angular separability in the feature space. That is, the larger the inter-class angle is, the lower the confusion will be. Based on this observation, we propose a novel loss function dubbed “Inter-Class Angular Loss” (ICAL), which explicitly models the class correlation and can be directly applied to many existing deep networks. By minimizing the proposed ICAL, the networks can effectively discriminate the examples in similar classes by enlarging the angle between their corresponding class vectors. Thorough experimental results on a series of vision and nonvision datasets confirm that ICAL critically improves the discriminative ability of various representative deep neural networks and generates superior performance to the original networks with conventional softmax loss.

scholarly journals Multiactivation Pooling Method in Convolutional Neural Networks for Image Recognition

2018 ◽  
Vol 2018 ◽  
pp. 1-15 ◽  
Author(s):  
Qi Zhao ◽  
Shuchang Lyu ◽  
Boxue Zhang ◽  
Wenquan Feng
Keyword(s):  
Neural Networks ◽  
Image Processing ◽  
Big Data ◽  
Convolutional Neural Networks ◽  
Image Recognition ◽  
Large Scale ◽  
Fog Computing ◽  
Feature Extractor ◽  
Benchmark Datasets ◽  
Classification Tasks

Convolutional neural networks (CNNs) are becoming more and more popular today. CNNs now have become a popular feature extractor applying to image processing, big data processing, fog computing, etc. CNNs usually consist of several basic units like convolutional unit, pooling unit, activation unit, and so on. In CNNs, conventional pooling methods refer to 2×2 max-pooling and average-pooling, which are applied after the convolutional or ReLU layers. In this paper, we propose a Multiactivation Pooling (MAP) Method to make the CNNs more accurate on classification tasks without increasing depth and trainable parameters. We add more convolutional layers before one pooling layer and expand the pooling region to 4×4, 8×8, 16×16, and even larger. When doing large-scale subsampling, we pick top-k activation, sum up them, and constrain them by a hyperparameter σ. We pick VGG, ALL-CNN, and DenseNets as our baseline models and evaluate our proposed MAP method on benchmark datasets: CIFAR-10, CIFAR-100, SVHN, and ImageNet. The classification results are competitive.

Discriminant Component Pruning: Regularization and Interpretation of Multilayered Backpropagation Networks

1999 ◽  
Vol 11 (3) ◽  
pp. 783-802 ◽  
Author(s):  
Randal A. Koene ◽  
Yoshio Takane
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Optimal Level ◽  
Neural Activation ◽  
Activation Functions ◽  
Large Networks ◽  
Pruning Algorithm ◽  
Generalization Performance ◽  
Reduced Rank ◽  
Classification Tasks

Neural networks are often employed as tools in classification tasks. The use of large networks increases the likelihood of the task's being learned, although it may also lead to increased complexity. Pruning is an effective way of reducing the complexity of large networks. We present discriminant components pruning (DCP), a method of pruning matrices of summed contributions between layers of a neural network. Attempting to interpret the underlying functions learned by the network can be aided by pruning the network. Generalization performance should be maintained at its optimal level following pruning. We demonstrate DCP's effectiveness at maintaining generalization performance, applicability to a wider range of problems, and the usefulness of such pruning for network interpretation. Possible enhancements are discussed for the identification of the optimal reduced rank and inclusion of nonlinear neural activation functions in the pruning algorithm.

scholarly journals Pashtu Numerals Recognition through Convolutional Neural Networks

2019 ◽  
pp. 91-96
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Character Recognition ◽  
Optical Character Recognition ◽  
Recognition Accuracy ◽  
Research Use ◽  
Optical Character ◽  
Classification Tasks ◽  
Scanned Images

In the proposed paper we introduce a new Pashtu numerals dataset having handwritten scanned images. We make the dataset publically available for scientific and research use. Pashtu language is used by more than fifty million people both for oral and written communication, but still no efforts are devoted to the Optical Character Recognition (OCR) system for Pashtu language. We introduce a new method for handwritten numerals recognition of Pashtu language through the deep learning based models. We use convolutional neural networks (CNNs) both for features extraction and classification tasks. We assess the performance of the proposed CNNs based model and obtained recognition accuracy of 91.45%.

scholarly journals Fish Recognition Model for Fraud Prevention using Convolutional Neural Networks

2021 ◽  
Author(s):  
Rhayane Monteiro ◽  
Morgana Ribeiro ◽  
Calebi Viana ◽  
Mario Wedney de Lima Moreira ◽  
Glacio Araújo ◽  
...  
Keyword(s):  
Neural Networks ◽  
Species Identification ◽  
Convolutional Neural Networks ◽  
Image Recognition ◽  
Fish Species ◽  
Effective Solution ◽  
Food Fraud ◽  
Fraud Prevention ◽  
Recognition Model ◽  
Classification Tasks

Abstract Fraud, misidentification, and adulteration of food, whether unintentional or purposeful, are a worldwide and growing concern. Aquaculture and fisheries are recognized as one of the sectors most vulnerable to food fraud. Besides, a series of risks related to health and distrust between consumer and popular market that this sector develop an effective solution for fraud control. Species identification is an essential aspect to expose commercial fraud. Convolutional neural networks (CNNs) are one of the most powerful tools for image recognition and classification tasks. Thus, the objective of this study is to propose a model of recognition of fish species based on CNNs. The results obtained show an algorithm with an accuracy of 86%, providing an effective solution to prevent fish fraud.

scholarly journals Behavioral Malware Detection Using Deep Graph Convolutional Neural Networks

2019 ◽  
Author(s):  
Angelo Schranko de Oliveira ◽  
Renato José Sassi
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Short Term Memory ◽  
Malware Detection ◽  
Detection Methods ◽  
New Public ◽  
Classification Tasks ◽  
Long Short Term Memory ◽  
Similar Area ◽  
Source Of Information

<div>Malware behavioral graphs provide a rich source of information that can be leveraged for detection and classification tasks. In this paper, we propose a novel behavioral malware detection method based on Deep Graph Convolutional Neural Networks (DGCNNs) to learn directly from API call sequences and their associated behavioral graphs. In order to train and evaluate the models, we created a new public domain dataset of more than 40,000 API call sequences resulting from the execution of malware and goodware instances in a sandboxed environment. Experimental results show that our models achieve similar Area Under the ROC Curve (AUC-ROC) and F1-Score to Long-Short Term Memory (LSTM) networks, widely used as the base architecture for behavioral malware detection methods, thus indicating that the models can effectively learn to distinguish between malicious and benign temporal patterns through convolution operations on graphs. To the best of our knowledge, this is the first paper that investigates the applicability of DGCNN to behavioral malware detection using API call sequences.</div>

scholarly journals Envision Foundational of Convolution Neural Network

2021 ◽  
Vol 10 (6) ◽  
pp. 54-60
Author(s):  
M Venkata Krishna Reddy* ◽  
Pradeep S.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Computer Graphics ◽  
Convolutional Neural Networks ◽  
Generative Models ◽  
Activation Functions ◽  
Machine Learning Classification ◽  
Network Intrusion ◽  
Recent Advances

1. Bilal, A. Jourabloo, M. Ye, X. Liu, and L. Ren. Do Convolutional Neural Networks Learn Class Hierarchy? IEEE Transactions on Visualization and Computer Graphics, 24(1):152–162, Jan. 2018. 2. M. Carney, B. Webster, I. Alvarado, K. Phillips, N. Howell, J. Griffith, J. Jongejan, A. Pitaru, and A. Chen. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20. ACM, Honolulu, HI, USA, 2020. 3. A. Karpathy. CS231n Convolutional Neural Networks for Visual Recognition, 2016 4. M. Kahng, N. Thorat, D. H. Chau, F. B. Viegas, and M. Wattenberg. GANLab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation. IEEE Transactions on Visualization and Computer Graphics, 25(1):310–320, Jan. 2019. 5. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015 6. M. Kahng, P. Y. Andrews, A. Kalro, and D. H. Chau. ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. IEEE Transactions on Visualization and Computer Graphics, 24(1):88–97, Jan. 2018. 7. https://cs231n.github.io/convolutional-networks/ 8. https://www.analyticsvidhya.com/blog/2020/02/learn-imageclassification-cnn-convolutional-neural-networks-3-datasets/ 9. https://towardsdatascience.com/understanding-cnn-convolutionalneural- network-69fd626ee7d4 10. https://medium.com/@birdortyedi_23820/deep-learning-lab-episode-2- cifar- 10-631aea84f11e 11. J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen. Recent advances in convolutional neural networks. Pattern Recognition, 77:354–377, May 2018. 12. Hamid, Y., Shah, F.A. and Sugumaram, M. (2014), ―Wavelet neural network model for network intrusion detection system‖, International Journal of Information Technology, Vol. 11 No. 2, pp. 251-263 13. G Sreeram , S Pradeep, K SrinivasRao , B.Deevan Raju , Parveen Nikhat , ― Moving ridge neuronal espionage network simulation for reticulum invasion sensing‖. International Journal of Pervasive Computing and Communications.https://doi.org/10.1108/IJPCC-05- 2020-0036 14. E. Stevens, L. Antiga, and T. Viehmann. Deep Learning with PyTorch. O’Reilly Media, 2019. 15. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015. 16. Aman Dureja, Payal Pahwa, ―Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks‖, Recent Advances in Computer Science , Vol 2, Issue 3, 2019 ,PP-156-161 17. https://missinglink.ai/guides/neural-network-concepts/7-types-neuralnetwork-activation-functions-right/

scholarly journals Spatio–Temporal Image Representation of 3D Skeletal Movements for View-Invariant Action Recognition with Deep Convolutional Neural Networks

Sensors ◽  
2019 ◽  
Vol 19 (8) ◽  
pp. 1932 ◽  
Author(s):  
Huy Hieu Pham ◽  
Houssam Salmane ◽  
Louahdi Khoudour ◽  
Alain Crouzil ◽  
Pablo Zegers ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Action Recognition ◽  
Large Scale ◽  
Image Representation ◽  
Human Action ◽  
Computational Time ◽  
Deep Convolutional Neural Networks ◽  
Classification Tasks ◽  
Spatio Temporal

Designing motion representations for 3D human action recognition from skeleton sequences is an important yet challenging task. An effective representation should be robust to noise, invariant to viewpoint changes and result in a good performance with low-computational demand. Two main challenges in this task include how to efficiently represent spatio–temporal patterns of skeletal movements and how to learn their discriminative features for classification tasks. This paper presents a novel skeleton-based representation and a deep learning framework for 3D action recognition using RGB-D sensors. We propose to build an action map called SPMF (Skeleton Posture-Motion Feature), which is a compact image representation built from skeleton poses and their motions. An Adaptive Histogram Equalization (AHE) algorithm is then applied on the SPMF to enhance their local patterns and form an enhanced action map, namely Enhanced-SPMF. For learning and classification tasks, we exploit Deep Convolutional Neural Networks based on the DenseNet architecture to learn directly an end-to-end mapping between input skeleton sequences and their action labels via the Enhanced-SPMFs. The proposed method is evaluated on four challenging benchmark datasets, including both individual actions, interactions, multiview and large-scale datasets. The experimental results demonstrate that the proposed method outperforms previous state-of-the-art approaches on all benchmark tasks, whilst requiring low computational time for training and inference.

Sign in / Sign up

Export Citation Format

Share Document