Discriminant Component Pruning: Regularization and Interpretation of Multilayered Backpropagation Networks

1999 ◽  
Vol 11 (3) ◽  
pp. 783-802 ◽  
Author(s):  
Randal A. Koene ◽  
Yoshio Takane
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Optimal Level ◽  
Neural Activation ◽  
Activation Functions ◽  
Large Networks ◽  
Pruning Algorithm ◽  
Generalization Performance ◽  
Reduced Rank ◽  
Classification Tasks

Neural networks are often employed as tools in classification tasks. The use of large networks increases the likelihood of the task's being learned, although it may also lead to increased complexity. Pruning is an effective way of reducing the complexity of large networks. We present discriminant components pruning (DCP), a method of pruning matrices of summed contributions between layers of a neural network. Attempting to interpret the underlying functions learned by the network can be aided by pruning the network. Generalization performance should be maintained at its optimal level following pruning. We demonstrate DCP's effectiveness at maintaining generalization performance, applicability to a wider range of problems, and the usefulness of such pruning for network interpretation. Possible enhancements are discussed for the identification of the optimal reduced rank and inclusion of nonlinear neural activation functions in the pruning algorithm.

Deep neural network based on generalized neo-fuzzy neurons and its learning based on backpropagation

2021 ◽  
Vol 26 (jai2021.26(1)) ◽  
pp. 32-41
Author(s):  
Bodyanskiy Y ◽  
◽  
Antonenko T ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Learning Process ◽  
Activation Function ◽  
Point Of View ◽  
Basic Unit ◽  
Theoretical Point ◽  
Activation Functions ◽  
Approximation Properties ◽  
Network Training

Modern approaches in deep neural networks have a number of issues related to the learning process and computational costs. This article considers the architecture grounded on an alternative approach to the basic unit of the neural network. This approach achieves optimization in the calculations and gives rise to an alternative way to solve the problems of the vanishing and exploding gradient. The main issue of the article is the usage of the deep stacked neo-fuzzy system, which uses a generalized neo-fuzzy neuron to optimize the learning process. This approach is non-standard from a theoretical point of view, so the paper presents the necessary mathematical calculations and describes all the intricacies of using this architecture from a practical point of view. From a theoretical point, the network learning process is fully disclosed. Derived all necessary calculations for the use of the backpropagation algorithm for network training. A feature of the network is the rapid calculation of the derivative for the activation functions of neurons. This is achieved through the use of fuzzy membership functions. The paper shows that the derivative of such function is a constant, and this is a reason for the statement of increasing in the optimization rate in comparison with neural networks which use neurons with more common activation functions (ReLU, sigmoid). The paper highlights the main points that can be improved in further theoretical developments on this topic. In general, these issues are related to the calculation of the activation function. The proposed methods cope with these points and allow approximation using the network, but the authors already have theoretical justifications for improving the speed and approximation properties of the network. The results of the comparison of the proposed network with standard neural network architectures are shown

scholarly journals Binary and Multiclass Text Classification by Means of Separable Convolutional Neural Network

Inventions ◽  
2021 ◽  
Vol 6 (4) ◽  
pp. 70
Author(s):  
Elena Solovyeva ◽  
Ali Abdullah
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Recurrent Neural Networks ◽  
Low Cost ◽  
Computational Cost ◽  
High Accuracy ◽  
Activation Functions ◽  
Fully Connected ◽  
Fully Connected Networks

In this paper, the structure of a separable convolutional neural network that consists of an embedding layer, separable convolutional layers, convolutional layer and global average pooling is represented for binary and multiclass text classifications. The advantage of the proposed structure is the absence of multiple fully connected layers, which is used to increase the classification accuracy but raises the computational cost. The combination of low-cost separable convolutional layers and a convolutional layer is proposed to gain high accuracy and, simultaneously, to reduce the complexity of neural classifiers. Advantages are demonstrated at binary and multiclass classifications of written texts by means of the proposed networks under the sigmoid and Softmax activation functions in convolutional layer. At binary and multiclass classifications, the accuracy obtained by separable convolutional neural networks is higher in comparison with some investigated types of recurrent neural networks and fully connected networks.

A Neural Networks Pruning and Data Fusion Based Intrusion Detection Model

2014 ◽  
Vol 651-653 ◽  
pp. 1772-1775
Author(s):  
Wei Gong
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Intrusion Detection ◽  
Data Fusion ◽  
Input Data ◽  
The Other ◽  
Pruning Algorithm ◽  
Detection Model ◽  
Process Ability

The abilities of summarization, learning and self-fitting and inner-parallel computing make artificial neural networks suitable for intrusion detection. On the other hand, data fusion based IDS has been used to solve the problem of distorting rate and failing-to-report rate and improve its performance. However, multi-sensor input-data makes the IDS lose its efficiency. The research of neural network based data fusion IDS tries to combine the strong process ability of neural network with the advantages of data fusion IDS. A neural network is designed to realize the data fusion and intrusion analysis and Pruning algorithm of neural networks is used for filtering information from multi-sensors. In the process of intrusion analysis pruning algorithm of neural networks is used for filtering information from multi-sensors so as to increase its performance and save the bandwidth of networks.

Comprehensive Overview of Neural Networks and Its Applications in Autonomous Vehicles

2019 ◽  
pp. 159-173
Author(s):  
Jay Rodge ◽  
Swati Jaiswal
Keyword(s):  
Neural Network ◽  
Artificial Intelligence ◽  
Neural Networks ◽  
Deep Learning ◽  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Learning Algorithms ◽  
Activation Functions ◽  
Comprehensive Overview ◽  
Unit Component

Deep learning and Artificial intelligence (AI) have been trending these days due to the capability and state-of-the-art results that they provide. They have replaced some highly skilled professionals with neural network-powered AI, also known as deep learning algorithms. Deep learning majorly works on neural networks. This chapter discusses about the working of a neuron, which is a unit component of neural network. There are numerous techniques that can be incorporated while designing a neural network, such as activation functions, training, etc. to improve its features, which will be explained in detail. It has some challenges such as overfitting, which are difficult to neglect but can be overcome using proper techniques and steps that have been discussed. The chapter will help the academician, researchers, and practitioners to further investigate the associated area of deep learning and its applications in the autonomous vehicle industry.

scholarly journals Adiabatic superconducting cells for ultra-low-power artificial neural networks

2016 ◽  
Vol 7 ◽  
pp. 1397-1403 ◽  
Author(s):  
Andrey E Schegolev ◽  
Nikolay V Klenov ◽  
Igor I Soloviev ◽  
Maxim V Tereshonok
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Low Power ◽  
Power Dissipation ◽  
Radial Basis Function Networks ◽  
Activation Functions ◽  
Ultra Low Power ◽  
Network Algorithms ◽  
Low Power Dissipation

We propose the concept of using superconducting quantum interferometers for the implementation of neural network algorithms with extremely low power dissipation. These adiabatic elements are Josephson cells with sigmoid- and Gaussian-like activation functions. We optimize their parameters for application in three-layer perceptron and radial basis function networks.

scholarly journals Interval universal approximation for neural networks

2022 ◽  
Vol 6 (POPL) ◽  
pp. 1-29
Author(s):  
Zi Wang ◽  
Aws Albarghouthi ◽  
Gautam Prakriya ◽  
Somesh Jha
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Approximation Problem ◽  
Activation Function ◽  
Constructive Proof ◽  
Universal Approximation ◽  
Activation Functions ◽  
Complete Problems ◽  
Robust Network ◽  
Interval Approximation

To verify safety and robustness of neural networks, researchers have successfully applied abstract interpretation , primarily using the interval abstract domain. In this paper, we study the theoretical power and limits of the interval domain for neural-network verification. First, we introduce the interval universal approximation (IUA) theorem. IUA shows that neural networks not only can approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using any well-behaved activation function, whose interval bounds are an arbitrarily close approximation of the set semantics of f (the result of applying f to a set of inputs). We call this notion of approximation interval approximation . Our theorem generalizes the recent result of Baader et al. from ReLUs to a rich class of activation functions that we call squashable functions . Additionally, the IUA theorem implies that we can always construct provably robust neural networks under ℓ ∞ -norm using almost any practical activation function. Second, we study the computational complexity of constructing neural networks that are amenable to precise interval analysis. This is a crucial question, as our constructive proof of IUA is exponential in the size of the approximation domain. We boil this question down to the problem of approximating the range of a neural network with squashable activation functions. We show that the range approximation problem (RA) is a Δ 2 -intermediate problem, which is strictly harder than NP -complete problems, assuming coNP ⊄ NP . As a result, IUA is an inherently hard problem : No matter what abstract domain or computational tools we consider to achieve interval approximation, there is no efficient construction of such a universal approximator. This implies that it is hard to construct a provably robust network, even if we have a robust network to start with.

scholarly journals Envision Foundational of Convolution Neural Network

2021 ◽  
Vol 10 (6) ◽  
pp. 54-60
Author(s):  
M Venkata Krishna Reddy* ◽  
Pradeep S.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Computer Graphics ◽  
Convolutional Neural Networks ◽  
Generative Models ◽  
Activation Functions ◽  
Machine Learning Classification ◽  
Network Intrusion ◽  
Recent Advances

1. Bilal, A. Jourabloo, M. Ye, X. Liu, and L. Ren. Do Convolutional Neural Networks Learn Class Hierarchy? IEEE Transactions on Visualization and Computer Graphics, 24(1):152–162, Jan. 2018. 2. M. Carney, B. Webster, I. Alvarado, K. Phillips, N. Howell, J. Griffith, J. Jongejan, A. Pitaru, and A. Chen. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20. ACM, Honolulu, HI, USA, 2020. 3. A. Karpathy. CS231n Convolutional Neural Networks for Visual Recognition, 2016 4. M. Kahng, N. Thorat, D. H. Chau, F. B. Viegas, and M. Wattenberg. GANLab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation. IEEE Transactions on Visualization and Computer Graphics, 25(1):310–320, Jan. 2019. 5. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015 6. M. Kahng, P. Y. Andrews, A. Kalro, and D. H. Chau. ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. IEEE Transactions on Visualization and Computer Graphics, 24(1):88–97, Jan. 2018. 7. https://cs231n.github.io/convolutional-networks/ 8. https://www.analyticsvidhya.com/blog/2020/02/learn-imageclassification-cnn-convolutional-neural-networks-3-datasets/ 9. https://towardsdatascience.com/understanding-cnn-convolutionalneural- network-69fd626ee7d4 10. https://medium.com/@birdortyedi_23820/deep-learning-lab-episode-2- cifar- 10-631aea84f11e 11. J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen. Recent advances in convolutional neural networks. Pattern Recognition, 77:354–377, May 2018. 12. Hamid, Y., Shah, F.A. and Sugumaram, M. (2014), ―Wavelet neural network model for network intrusion detection system‖, International Journal of Information Technology, Vol. 11 No. 2, pp. 251-263 13. G Sreeram , S Pradeep, K SrinivasRao , B.Deevan Raju , Parveen Nikhat , ― Moving ridge neuronal espionage network simulation for reticulum invasion sensing‖. International Journal of Pervasive Computing and Communications.https://doi.org/10.1108/IJPCC-05- 2020-0036 14. E. Stevens, L. Antiga, and T. Viehmann. Deep Learning with PyTorch. O’Reilly Media, 2019. 15. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015. 16. Aman Dureja, Payal Pahwa, ―Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks‖, Recent Advances in Computer Science , Vol 2, Issue 3, 2019 ,PP-156-161 17. https://missinglink.ai/guides/neural-network-concepts/7-types-neuralnetwork-activation-functions-right/

scholarly journals Odia Characters and Numerals Recognition using Hopfield Neural Network Based on Zoning Feature

2019 ◽  
Vol 8 (2) ◽  
pp. 4928-4937 ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Character Recognition ◽  
Optical Character Recognition ◽  
Hopfield Neural Network ◽  
Hopfield Network ◽  
Research Areas ◽  
Different Types ◽  
Classification Tasks ◽  
Text Version

Odia character and digits recognition area are vital issues of these days in computer vision. In this paper a Hope field neural network design to solve the printed Odia character recognition has been discussed. Optical Character Recognition (OCR) is the principle of applying conversion of the pictures from handwritten, printed or typewritten to machine encoded text version. Artificial Neural Networks (ANNs) trained as a classifier and it had been trained, supported the rule of Hopfield Network by exploitation code designed within the MATLAB. Preprocessing of data (image acquisition, binarization, skeletonization, skew detection and correction, image cropping, resizing, implementation and digitalization) all these activities have been carried out using MATLAB. The OCR, designed a number of the thought accuses non-standard speech for different types of languages. Segmentation, feature extraction, classification tasks is the well-known techniques for reviewing of Odia characters and outlined with their weaknesses, relative strengths. It is expected that who are interested to figure within the field of recognition of Odia characters are described in this paper. Recognition of Odia printed characters, numerals, machine characters of research areas finds costly applications within the banks, industries, offices. In this proposed work we devolve an efficient and robust mechanism in which Odia characters are recognized by the Hopfield Neural Networks (HNN).

scholarly journals A Novel Architecture to Classify Histopathology Images Using Convolutional Neural Networks

2020 ◽  
Vol 10 (8) ◽  
pp. 2929 ◽  
Author(s):  
Ibrahem Kandel ◽  
Mauro Castelli
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
State Of The Art ◽  
Treatment Plan ◽  
Tissue Structure ◽  
Activation Functions ◽  
Proposed Model ◽  
Histopathology Images ◽  
Fully Connected

Histopathology is the study of tissue structure under the microscope to determine if the cells are normal or abnormal. Histopathology is a very important exam that is used to determine the patients’ treatment plan. The classification of histopathology images is very difficult to even an experienced pathologist, and a second opinion is often needed. Convolutional neural network (CNN), a particular type of deep learning architecture, obtained outstanding results in computer vision tasks like image classification. In this paper, we propose a novel CNN architecture to classify histopathology images. The proposed model consists of 15 convolution layers and two fully connected layers. A comparison between different activation functions was performed to detect the most efficient one, taking into account two different optimizers. To train and evaluate the proposed model, the publicly available PatchCamelyon dataset was used. The dataset consists of 220,000 annotated images for training and 57,000 unannotated images for testing. The proposed model achieved higher performance compared to the state-of-the-art architectures with an AUC of 95.46%.

Hierarchical Function Approximation with a Neural Network Model

2012 ◽  
pp. 466-479
Author(s):  
Luis F. de Mingo ◽  
Nuria Gómez ◽  
Fernando Arroyo ◽  
Juan Castellanos
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Model ◽  
Neural Network Model ◽  
Polynomial Equation ◽  
Neural System ◽  
Input Pattern ◽  
Activation Functions ◽  
Proposed Model ◽  
Generate Polynomial

This article presents a neural network model that permits to build a conceptual hierarchy to approximate functions over a given interval. Bio-inspired axo-axonic connections are used. In these connections the signal weight between two neurons is computed by the output of other neuron. Such arquitecture can generate polynomial expressions with lineal activation functions. This network can approximate any pattern set with a polynomial equation. This neural system classifies an input pattern as an element belonging to a category that the system has, until an exhaustive classification is obtained. The proposed model is not a hierarchy of neural networks, it establishes relationships among all the different neural networks in order to propagate the activation. Each neural network is in charge of the input pattern recognition to any prototyped category, and also in charge of transmitting the activation to other neural networks to be able to continue with the approximation.

Sign in / Sign up

Export Citation Format

Share Document