scholarly journals Neural network interpretation using descrambler groups

2021 ◽  
Vol 118 (5) ◽  
pp. e2016917118
Author(s):  
Jake L. Amey ◽  
Jake Keeley ◽  
Tajwar Choudhury ◽  
Ilya Kuprov
Keyword(s):  
Harmonic Functions ◽  
Deep Neural Networks ◽  
Notch Filter ◽  
Digital Signal ◽  
Initial Guess ◽  
Neural Nets ◽  
Spectral Filtering ◽  
Spin Resonance ◽  
Fully Connected ◽  
Theoretical Procedure

The lack of interpretability and trust is a much-criticized feature of deep neural networks. In fully connected nets, the signaling between inner layers is scrambled because backpropagation training does not require perceptrons to be arranged in any particular order. The result is a black box; this problem is particularly severe in scientific computing and digital signal processing (DSP), where neural nets perform abstract mathematical transformations that do not reduce to features or concepts. We present here a group-theoretical procedure that attempts to bring inner-layer signaling into a human-readable form, the assumption being that this form exists and has identifiable and quantifiable features—for example, smoothness or locality. We applied the proposed method to DEERNet (a DSP network used in electron spin resonance) and managed to descramble it. We found considerable internal sophistication: the network spontaneously invents a bandpass filter, a notch filter, a frequency axis rescaling transformation, frequency-division multiplexing, group embedding, spectral filtering regularization, and a map from harmonic functions into Chebyshev polynomials—in 10 min of unattended training from a random initial guess.

scholarly journals Ship-Iceberg Classification in SAR and Multispectral Satellite Images with Neural Networks

2020 ◽  
Vol 12 (15) ◽  
pp. 2353
Author(s):  
Henning Heiselberg
Keyword(s):  
Neural Networks ◽  
Satellite Images ◽  
Deep Neural Networks ◽  
Data Augmentation ◽  
Neural Nets ◽  
The Arctic ◽  
Support Vector ◽  
Sar Images ◽  
Multispectral Satellite Images ◽  
The Cost

Classification of ships and icebergs in the Arctic in satellite images is an important problem. We study how to train deep neural networks for improving the discrimination of ships and icebergs in multispectral satellite images. We also analyze synthetic-aperture radar (SAR) images for comparison. The annotated datasets of ships and icebergs are collected from multispectral Sentinel-2 data and taken from the C-CORE dataset of Sentinel-1 SAR images. Convolutional Neural Networks with a range of hyperparameters are tested and optimized. Classification accuracies are considerably better for deep neural networks than for support vector machines. Deeper neural nets improve the accuracy per epoch but at the cost of longer processing time. Extending the datasets with semi-supervised data from Greenland improves the accuracy considerably whereas data augmentation by rotating and flipping the images has little effect. The resulting classification accuracies for ships and icebergs are 86% for the SAR data and 96% for the MSI data due to the better resolution and more multispectral bands. The size and quality of the datasets are essential for training the deep neural networks, and methods to improve them are discussed. The reduced false alarm rates and exploitation of multisensory data are important for Arctic search and rescue services.

scholarly journals Accelerating Deep Neural Networks by Combining Block-Circulant Matrices and Low-Precision Weights

Electronics ◽  
2019 ◽  
Vol 8 (1) ◽  
pp. 78 ◽  
Author(s):  
Zidi Qin ◽  
Di Zhu ◽  
Xingwei Zhu ◽  
Xuan Chen ◽  
Yinghuan Shi ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Circulant Matrix ◽  
Circulant Matrices ◽  
Matrix Vector Multiplication ◽  
Processing Power ◽  
Measurement Results ◽  
Block Circulant Matrix ◽  
Storage Complexity ◽  
Fully Connected

As a key ingredient of deep neural networks (DNNs), fully-connected (FC) layers are widely used in various artificial intelligence applications. However, there are many parameters in FC layers, so the efficient process of FC layers is restricted by memory bandwidth. In this paper, we propose a compression approach combining block-circulant matrix-based weight representation and power-of-two quantization. Applying block-circulant matrices in FC layers can reduce the storage complexity from O ( k 2 ) to O ( k ) . By quantizing the weights into integer powers of two, the multiplications in the reference can be replaced by shift and add operations. The memory usages of models for MNIST, CIFAR-10 and ImageNet can be compressed by 171 × , 2731 × and 128 × with minimal accuracy loss, respectively. A configurable parallel hardware architecture is then proposed for processing the compressed FC layers efficiently. Without multipliers, a block matrix-vector multiplication module (B-MV) is used as the computing kernel. The architecture is flexible to support FC layers of various compression ratios with small footprint. Simultaneously, the memory access can be significantly reduced by using the configurable architecture. Measurement results show that the accelerator has a processing power of 409.6 GOPS, and achieves 5.3 TOPS/W energy efficiency at 800 MHz.

scholarly journals Detection and control of a gyroscopically induced vibration to improve the balance of a single-wheel robot

2017 ◽  
Vol 37 (3) ◽  
pp. 443-455 ◽  
Author(s):  
Sangdeok Lee ◽  
Seul Jung
Keyword(s):  
Digital Signal Processor ◽  
High Speed ◽  
Balance Control ◽  
A Priori ◽  
Experimental Studies ◽  
Notch Filter ◽  
Digital Signal ◽  
Control Performance ◽  
And Control ◽  
Balancing Control

In this article, an experimental investigation of the detection of a gyroscopically induced vibration and the balancing control performance of a single-wheel robot is presented. The balance of the single-wheel robot was intended to be maintained by virtue of the gyroscopic effect induced from a highly rotating flywheel. Since the flywheel rotates at a high speed, an asymmetrical structure of a flywheel causes an irregular rotation and becomes one of the major vibration sources. A vibration was detected and suppressed a priori before applying control algorithms to the robot. Gyroscopically induced vibrations can empirically be detected with different rotational velocities. The detection of the balancing angle of the single-wheel robot was accomplished by using an attitude and heading reference system. After identifying the vibrating frequencies, a notch filter was designed to suppress the vibration at the typical frequencies identified through experiments. A digital filter was designed and implemented in a digital signal processor(DSP) along with the control scheme for the balance control performance. The performance of the proposed method was verified by the experimental studies on the balancing control of the single-wheel robot. Experimental results confirmed that the notch filter designed following the detection of the flywheel’s vibration actually improved the balancing control performance. A half of the vibration magnitude was reduced by the proposal.

scholarly journals A HYBRID MODEL USING THE PRETRAINED BERT AND DEEP NEURAL NETWORKS WITH RICH FEATURE FOR EXTRACTIVE TEXT SUMMARIZATION

2021 ◽  
Vol 37 (2) ◽  
pp. 123-143
Author(s):  
Tuan Minh Luu ◽  
Huong Thanh Le ◽  
Tan Minh Hoang
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Text Summarization ◽  
Training Dataset ◽  
Extractive Summarization ◽  
Input Text ◽  
Summarization System ◽  
Fully Connected

Deep neural networks have been applied successfully to extractive text summarization tasks with the accompany of large training datasets. However, when the training dataset is not large enough, these models reveal certain limitations that affect the quality of the system’s summary. In this paper, we propose an extractive summarization system basing on a Convolutional Neural Network and a Fully Connected network for sentence selection. The pretrained BERT multilingual model is used to generate embeddings vectors from the input text. These vectors are combined with TF-IDF values to produce the input of the text summarization system. Redundant sentences from the output summary are eliminated by the Maximal Marginal Relevance method. Our system is evaluated with both English and Vietnamese languages using CNN and Baomoi datasets, respectively. Experimental results show that our system achieves better results comparing to existing works using the same dataset. It confirms that our approach can be effectively applied to summarize both English and Vietnamese languages.

scholarly journals Regularizing Deep Neural Networks with an Ensemble-based Decorrelation Method

2018 ◽  
Author(s):  
Shuqin Gu ◽  
Yuexian Hou ◽  
Lipeng Zhang ◽  
Yazhou Zhang
Keyword(s):  
Neural Networks ◽  
Ensemble Learning ◽  
Convolutional Neural Networks ◽  
Deep Neural Networks ◽  
Experimental Results ◽  
Excellent Performance ◽  
Hidden Layer ◽  
Base Learner ◽  
Fully Connected

Although Deep Neural Networks (DNNs) have achieved excellent performance in many tasks, improving the generalization capacity of DNNs still remains a challenge. In this work, we propose a novel regularizer named Ensemble-based Decorrelation Method (EDM), which is motivated by the idea of the ensemble learning to improve generalization capacity of DNNs. EDM can be applied to hidden layers in fully connected neural networks or convolutional neural networks. We treat each hidden layer as an ensemble of several base learners through dividing all the hidden units into several non-overlap groups, and each group will be viewed as a base learner. EDM encourages DNNs to learn more diverse representations by minimizing the covariance between all base learners during the training step. Experimental results on MNIST and CIFAR datasets demonstrate that EDM can effectively reduce the overfitting and improve the generalization capacity of DNNs  

scholarly journals FNN and Auto Encoder Deep Learning-Based Algorithm for Android Cyber Security

2020 ◽  
Vol 8 (5) ◽  
pp. 3292-3296
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Detection System ◽  
Malware Detection ◽  
Cyber Attacks ◽  
Open Architecture ◽  
Android Malware ◽  
Android Malware Detection ◽  
Fully Connected

Android is susceptible to malware attacks due to its open architecture, large user base and access to its code. Mobile or android malware attacks are increasing from last year. These are common threats for every internet-accessible device. From Researchers Point of view 50% increase in cyber-attacks targeting Android Mobile phones since last year. Malware attackers increasingly turning their attention to attacking smartphones with credential-theft, surveillance, and malicious advertising. Security investigation in the android mobile system has relied on analysis for malware or threat detection using binary samples or system calls with behavior profile for malicious applications is generated and then analyzed. The resulting report is then used to detect android application malware or threats using manual features. To dispose of malicious applications in the mobile device, we propose an Android malware detection system using deep learning techniques which gives security for mobile or android. FNN(Fully-connected FeedForward Deep Neural Networks) and AutoEncoder algorithm from deep learning provide Extensive experiments on a real-world dataset that reaches to an accuracy of 95 %. These papers explain Deep learning FNN(Fully-connected FeedForward Deep Neural Networks) and AutoEncoder approach for android malware detection.

scholarly journals A Smoothed LASSO Based DNN Sparsification Technique

2020 ◽  
Author(s):  
Nitin Chandrachoodan ◽  
Basava Naga Girish Koneru ◽  
Vinita Vasudevan
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Maximum Error ◽  
Pruning Algorithm ◽  
Operator Functions ◽  
Selection Operator ◽  
Fully Connected

<div>Deep Neural Networks (DNNs) are increasingly being used in a variety of applications. However, DNNs have huge computational and memory requirements. One way to reduce these requirements is to sparsify DNNs by using smoothed LASSO (Least Absolute Shrinkage and Selection Operator) functions. In this paper, we show that for the same maximum error with respect to the LASSO function, the sparsity values obtained using various smoothed LASSO functions are similar. We also propose a layer-wise DNN pruning algorithm, where the layers are pruned based on their individual allocated accuracy loss budget determined by estimates of the reduction in number of multiply-accumulate operations (in convolutional layers) and weights (in fully connected layers). Further, the structured LASSO variants in both convolutional and fully connected layers are explored within the smoothed LASSO framework and the tradeoffs involved are discussed. The efficacy of proposed algorithm in enhancing the sparsity within the allowed degradation in DNN accuracy and results obtained on structured LASSO variants are shown on MNIST, SVHN, CIFAR-10, and Imagenette datasets.</div>

scholarly journals Notes on the Symmetries of 2-Layer ReLU-Networks

2020 ◽  
Vol 1 ◽  
pp. 6
Author(s):  
Henning Petzka ◽  
Martin Trimmel ◽  
Cristian Sminchisescu
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Complete Characterization ◽  
Activation Functions ◽  
Network Function ◽  
Fully Connected ◽  
Fully Connected Networks

Symmetries in neural networks allow different weight configurations leading to the same network function. For odd activation functions, the set of transformations mapping between such configurations have been studied extensively, but less is known for neural networks with ReLU activation functions. We give a complete characterization for fully-connected networks with two layers. Apart from two well-known transformations, only degenerated situations allow additional transformations that leave the network function unchanged. Reduction steps can remove only part of the degenerated cases. Finally, we present a non-degenerate situation for deep neural networks leading to new transformations leaving the network function intact.

Sign in / Sign up

Export Citation Format

Share Document