Evolving Node Transfer Functions in Deep Neural Networks for Pattern Recognition

2017 ◽  
pp. 206-215
Author(s):  
Dmytro Vodianyk ◽  
Przemysław Rokita
Keyword(s):  
Neural Networks ◽  
Pattern Recognition ◽  
Deep Neural Networks ◽  
Transfer Functions

scholarly journals Memristor crossbar architectures for implementing deep neural networks

2021 ◽  
Author(s):  
Xiaoyang Liu ◽  
Zhigang Zeng
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Pattern Recognition ◽  
Deep Neural Networks ◽  
Activation Functions ◽  
Simulation Results ◽  
Memristor Crossbars ◽  
Memristor Crossbar ◽  
Fully Connected

AbstractThe paper presents memristor crossbar architectures for implementing layers in deep neural networks, including the fully connected layer, the convolutional layer, and the pooling layer. The crossbars achieve positive and negative weight values and approximately realize various nonlinear activation functions. Then the layers constructed by the crossbars are adopted to build the memristor-based multi-layer neural network (MMNN) and the memristor-based convolutional neural network (MCNN). Two kinds of in-situ weight update schemes, which are the fixed-voltage update and the approximately linear update, respectively, are used to train the networks. Consider variations resulted from the inherent characteristics of memristors and the errors of programming voltages, the robustness of MMNN and MCNN to these variations is analyzed. The simulation results on standard datasets show that deep neural networks (DNNs) built by the memristor crossbars work satisfactorily in pattern recognition tasks and have certain robustness to memristor variations.

scholarly journals Deep neural networks – a developmental perspective

2016 ◽  
Vol 5 ◽  
Author(s):  
Biing Hwang Juang
Keyword(s):  
Neural Networks ◽  
Pattern Recognition ◽  
Deep Neural Networks ◽  
Developmental Perspective ◽  
Performance Improvements ◽  
Research Activities ◽  
Statistical Pattern ◽  
Significant Performance ◽  
The Subject ◽  
Learning Guide

There is a recent surge in research activities around “deep neural networks” (DNN). While the notion of neural networks have enjoyed cycles of enthusiasm, which may continue its ebb and flow, concrete advances now abound. Significant performance improvements have been shown in a number of pattern recognition tasks. As a technical topic, DNN is important in classes and tutorial articles and related learning resources are available. Streams of questions, nonetheless, never subside from students or researchers and there appears to be a frustrating tendency among the learners to treat DNN simply as a black box. This is an awkward and alarming situation in education. This paper thus has the intent to help the reader to properly understand DNN, not just its mechanism (what and how) but its motivation and justification (why). It is written from a developmental perspective with a comprehensive view, from the very basic but oft-forgotten principle of statistical pattern recognition and decision theory, through the problem stages that may be encountered during system design, to key ideas that led to the new advance. This paper can serve as a learning guide with historical reviews and important references, helpful in reaching an insightful understanding of the subject.

scholarly journals Information Bottleneck Theory Based Exploration of Cascade Learning

Entropy ◽  
2021 ◽  
Vol 23 (10) ◽  
pp. 1360
Author(s):  
Xin Du ◽  
Katayoun Farrahi ◽  
Mahesan Niranjan
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Pattern Recognition ◽  
Mutual Information ◽  
Theoretical Approach ◽  
Deep Neural Networks ◽  
Layer By Layer ◽  
Information Compression ◽  
Excellent Performance ◽  
Information Bottleneck

In solving challenging pattern recognition problems, deep neural networks have shown excellent performance by forming powerful mappings between inputs and targets, learning representations (features) and making subsequent predictions. A recent tool to help understand how representations are formed is based on observing the dynamics of learning on an information plane using mutual information, linking the input to the representation (I(X;T)) and the representation to the target (I(T;Y)). In this paper, we use an information theoretical approach to understand how Cascade Learning (CL), a method to train deep neural networks layer-by-layer, learns representations, as CL has shown comparable results while saving computation and memory costs. We observe that performance is not linked to information–compression, which differs from observation on End-to-End (E2E) learning. Additionally, CL can inherit information about targets, and gradually specialise extracted features layer-by-layer. We evaluate this effect by proposing an information transition ratio, I(T;Y)/I(X;T), and show that it can serve as a useful heuristic in setting the depth of a neural network that achieves satisfactory accuracy of classification.

scholarly journals ExpDNN: Explainable Deep Neural Network

2021 ◽  
Author(s):  
Chi-Hua Chen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Pattern Recognition ◽  
Feature Extraction ◽  
High Performance ◽  
Deep Neural Network ◽  
Deep Neural Networks ◽  
Regression Method ◽  
Linear Regression Method ◽  
Absolute Value

Abstract In recent years, deep neural networks have been applied to obtain high performance of prediction, classification, and pattern recognition. However, the weights in these deep neural networks are difficult to be explained. Although a linear regression method can provide explainable results, the method is not suitable in the case of input interaction. Therefore, an explainable deep neural network (ExpDNN) with explainable layers is proposed to obtain explainable results in the case of input interaction. Three cases were given to evaluate the proposed ExpDNN, and the results showed that the absolute value of weight in an explainable layer can be used to explain the weight of corresponding input for feature extraction.

Sign in / Sign up

Export Citation Format

Share Document