scholarly journals Study of Quantized Hardware Deep Neural Networks Based on Resistive Switching Devices, Conventional versus Convolutional Approaches

Electronics ◽  
2021 ◽  
Vol 10 (3) ◽  
pp. 346
Author(s):  
Rocío Romero-Zaliz ◽  
Eduardo Pérez ◽  
Francisco Jiménez-Molinos ◽  
Christian Wenger ◽  
Juan B. Roldán
Keyword(s):  
Neural Networks ◽  
Network Topology ◽  
Deep Neural Networks ◽  
Hardware Implementation ◽  
Image Features ◽  
Comprehensive Analysis ◽  
Image Databases ◽  
Data Complexity ◽  
Conductance Quantization ◽  
Hidden Layer

A comprehensive analysis of two types of artificial neural networks (ANN) is performed to assess the influence of quantization on the synaptic weights. Conventional multilayer-perceptron (MLP) and convolutional neural networks (CNN) have been considered by changing their features in the training and inference contexts, such as number of levels in the quantization process, the number of hidden layers on the network topology, the number of neurons per hidden layer, the image databases, the number of convolutional layers, etc. A reference technology based on 1T1R structures with bipolar memristors including HfO2 dielectrics was employed, accounting for different multilevel schemes and the corresponding conductance quantization algorithms. The accuracy of the image recognition processes was studied in depth. This type of studies are essential prior to hardware implementation of neural networks. The obtained results support the use of CNNs for image domains. This is linked to the role played by convolutional layers at extracting image features and reducing the data complexity. In this case, the number of synaptic weights can be reduced in comparison to MLPs.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

scholarly journals Hardware implementation of radial-basis neural networks with Gaussian activation functions on FPGA

2021 ◽  
Author(s):  
Volodymyr Shymkovych ◽  
Sergii Telenyk ◽  
Petro Kravets
Keyword(s):  
Neural Networks ◽  
Hardware Implementation ◽  
Gaussian Function ◽  
Activation Function ◽  
Rbf Neural Networks ◽  
Activation Functions ◽  
Rbf Network ◽  
Combination Scheme ◽  
Radial Basis ◽  
Hidden Layer

AbstractThis article introduces a method for realizing the Gaussian activation function of radial-basis (RBF) neural networks with their hardware implementation on field-programmable gaits area (FPGAs). The results of modeling of the Gaussian function on FPGA chips of different families have been presented. RBF neural networks of various topologies have been synthesized and investigated. The hardware component implemented by this algorithm is an RBF neural network with four neurons of the latent layer and one neuron with a sigmoid activation function on an FPGA using 16-bit numbers with a fixed point, which took 1193 logic matrix gate (LUTs—LookUpTable). Each hidden layer neuron of the RBF network is designed on an FPGA as a separate computing unit. The speed as a total delay of the combination scheme of the block RBF network was 101.579 ns. The implementation of the Gaussian activation functions of the hidden layer of the RBF network occupies 106 LUTs, and the speed of the Gaussian activation functions is 29.33 ns. The absolute error is ± 0.005. The Spartan 3 family of chips for modeling has been used to get these results. Modeling on chips of other series has been also introduced in the article. RBF neural networks of various topologies have been synthesized and investigated. Hardware implementation of RBF neural networks with such speed allows them to be used in real-time control systems for high-speed objects.

scholarly journals Regularizing Deep Neural Networks with an Ensemble-based Decorrelation Method

2018 ◽  
Author(s):  
Shuqin Gu ◽  
Yuexian Hou ◽  
Lipeng Zhang ◽  
Yazhou Zhang
Keyword(s):  
Neural Networks ◽  
Ensemble Learning ◽  
Convolutional Neural Networks ◽  
Deep Neural Networks ◽  
Experimental Results ◽  
Excellent Performance ◽  
Hidden Layer ◽  
Base Learner ◽  
Fully Connected

Although Deep Neural Networks (DNNs) have achieved excellent performance in many tasks, improving the generalization capacity of DNNs still remains a challenge. In this work, we propose a novel regularizer named Ensemble-based Decorrelation Method (EDM), which is motivated by the idea of the ensemble learning to improve generalization capacity of DNNs. EDM can be applied to hidden layers in fully connected neural networks or convolutional neural networks. We treat each hidden layer as an ensemble of several base learners through dividing all the hidden units into several non-overlap groups, and each group will be viewed as a base learner. EDM encourages DNNs to learn more diverse representations by minimizing the covariance between all base learners during the training step. Experimental results on MNIST and CIFAR datasets demonstrate that EDM can effectively reduce the overfitting and improve the generalization capacity of DNNs  

scholarly journals Deep neural network: Recognize Data Management of Artificial Intelligence in Retail

2019 ◽  
Vol 8 (10) ◽  
pp. 3594-3596
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Data Management ◽  
Deep Neural Networks ◽  
Data Management System ◽  
Research Approach ◽  
Vast Amount ◽  
Retail Business ◽  
Combining Data ◽  
Hidden Layer

Deep neural networks with the artificial intelligence on Machine Learning (ML) algorithms constitute the best design specifically to deal with vast amount of data for retail business. The limited research approach is referred towards reducing memory consumption on integrating ML algorithms on data management system. This paper proposed combining data management and deep neural networks, ideas to build systems, which vast amount data can share in the database system. Therefore, ML algorithm has a pattern with multi-hidden layer that can use to synthesis different decision within a minimum processing. Finally, system precede and follow a NoSQL layers of a model employs in-memory database compression techniques and executes data management challenges with large datasets successfully.

scholarly journals Emergence of Network Motifs in Deep Neural Networks

Entropy ◽  
2020 ◽  
Vol 22 (2) ◽  
pp. 204
Author(s):  
Matteo Zambra ◽  
Amos Maritan ◽  
Alberto Testolin
Keyword(s):  
Neural Networks ◽  
Network Topology ◽  
Deep Neural Networks ◽  
Network Science ◽  
Network Motifs ◽  
Structural And Functional Properties ◽  
Topological Modules ◽  
Weight Initialization ◽  
Learning Principles ◽  
Fully Connected

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.

scholarly journals Deep neural networks for rotation-invariance approximation and learning

2019 ◽  
Vol 17 (05) ◽  
pp. 737-772 ◽  
Author(s):  
Charles K. Chui ◽  
Shao-Bo Lin ◽  
Ding-Xuan Zhou
Keyword(s):  
Neural Networks ◽  
Function Approximation ◽  
Deep Neural Networks ◽  
Rotation Invariance ◽  
Approximation Accuracy ◽  
Radial Functions ◽  
Euclidian Space ◽  
Learning Capabilities ◽  
Optimal Function ◽  
Hidden Layer

Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high-dimensional Euclidian space. It is shown that deep nets have much better performance than shallow nets (with only one hidden layer) in terms of approximation accuracy and learning capabilities. In particular, for learning radial functions, it is shown that near-optimal rate can be achieved by deep nets but not by shallow nets. Our results illustrate the necessity of depth in neural network design for realization of rotation-invariance target functions.

scholarly journals Detection of Melanoma Skin Cancer with Deep Neural Networks

2019 ◽  
Vol 4 (4) ◽  
Keyword(s):  
Neural Networks ◽  
Skin Cancer ◽  
High Performance ◽  
Deep Neural Networks ◽  
Histopathological Examination ◽  
Skin Lesions ◽  
Image Features ◽  
Test Accuracy ◽  
Data Sets ◽  
Deep Convolutional Neural Networks

Detection of skin cancer involves several steps of examinations first being visual diagnosis that is followed by dermoscopic analysis, a biopsy, and histopathological examination. The classification of skin lesions in the first step is critical and challenging as classes vary by minute appearance in skin lesions. Deep convolutional neural networks (CNNs) have great potential in multicategory image-based classification by considering coarse-to-fine image features. This study aims to demonstrate how to classify skin lesions, in particular, melanoma, using CNN trained on data sets with disease labels. We developed and trained our own CNN model using a subset of the images from International Skin Imaging Collaboration (ISIC) Dermoscopic Archive. To test the performance of the proposed model, we used a different subset of images from the same archive as the test set. Our model is trained to classify images into two categories: malignant melanoma and nevus and is shown to achieve excellent classification results with high test accuracy (91.16%) and high performance as measured by various metrics. Our study demonstrated the potential of using deep neural networks to assist early detection of melanoma and thereby improve the patient survival rate from this aggressive skin cancer.

scholarly journals Towards a high robust neural network via feature matching

2021 ◽  
Author(s):  
Jian Li ◽  
Yanming Guo ◽  
Songyang Lao ◽  
Yulun Wu ◽  
Liang Bai ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Feature Matching ◽  
Feature Vector ◽  
State Of The Art ◽  
Model Performance ◽  
Image Features ◽  
Classification Systems ◽  
Adversarial Attack

AbstractImage classification systems have been found vulnerable to adversarial attack, which is imperceptible to human but can easily fool deep neural networks. Recent researches indicate that regularizing the network by introducing randomness could greatly improve the model’s robustness against adversarial attack, but the randomness module would normally involve complex calculations and numerous additional parameters and seriously affect the model performance on clean data. In this paper, we propose a feature matching module to regularize the network. Specifically, our model learns a feature vector for each category and imposes additional restrictions on image features. Then, the similarity between image features and category features is used as the basis for classification. Our method does not introduce any additional network parameters than undefended model and can be easily integrated into any neural network. Experiments on the CIFAR10 and SVHN datasets highlight that our proposed module can effectively improve both clean data and perturbed data accuracy in comparison with the state-of-the-art defense methods and outperform the L2P method by 6.3$$\%$$ % , 24$$\%$$ % on clean and perturbed data, respectively, using ResNet-V2(18) architecture.

scholarly journals Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustness

2019 ◽  
Author(s):  
NhatHai Phan ◽  
Minh N. Vu ◽  
Yang Liu ◽  
Ruoming Jin ◽  
Dejing Dou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Theoretical Analysis ◽  
Gaussian Noise ◽  
Deep Neural Networks ◽  
Differential Privacy ◽  
Trade Off ◽  
Adversarial Examples ◽  
Hidden Layer ◽  
Privacy Budget

In this paper, we propose a novel Heterogeneous Gaussian Mechanism (HGM) to preserve differential privacy in deep neural networks, with provable robustness against adversarial examples. We first relax the constraint of the privacy budget in the traditional Gaussian Mechanism from (0, 1] to (0, infty), with a new bound of the noise scale to preserve differential privacy. The noise in our mechanism can be arbitrarily redistributed, offering a distinctive ability to address the trade-off between model utility and privacy loss. To derive provable robustness, our HGM is applied to inject Gaussian noise into the first hidden layer. Then, a tighter robustness bound is proposed. Theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of differentially private deep neural networks, compared with baseline approaches, under a variety of model attacks.

Hardware Implementation of Sign Language to Text Converter Using Deep Neural Networks

2020 ◽  
Author(s):  
Harsha Vardhan Guda ◽  
Srivenkat Guntur ◽  
Gowri Pratyusha M ◽  
Kunal Gupta ◽  
Priyanka Volam ◽  
...  
Keyword(s):  
Neural Networks ◽  
Sign Language ◽  
Deep Neural Networks ◽  
Hardware Implementation
Sign in / Sign up

Export Citation Format

Share Document