scholarly journals Communication Failure Resilient Distributed Neural Network for Edge Devices

Electronics ◽  
2021 ◽  
Vol 10 (14) ◽  
pp. 1614
Author(s):  
Jonghun Jeong ◽  
Jong Sung Park ◽  
Hoeseok Yang
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
High Performance ◽  
State Of The Art ◽  
Wearable Devices ◽  
Communication Failure ◽  
Canadian Institute ◽  
Multiple Devices ◽  
Knowledge Distillation ◽  
Partitioning Technique

Recently, the necessity to run high-performance neural networks (NN) is increasing even in resource-constrained embedded systems such as wearable devices. However, due to the high computational and memory requirements of the NN applications, it is typically infeasible to execute them on a single device. Instead, it has been proposed to run a single NN application cooperatively on top of multiple devices, a so-called distributed neural network. In the distributed neural network, workloads of a single big NN application are distributed over multiple tiny devices. While the computation overhead could effectively be alleviated by this approach, the existing distributed NN techniques, such as MoDNN, still suffer from large traffics between the devices and vulnerability to communication failures. In order to get rid of such big communication overheads, a knowledge distillation based distributed NN, called Network of Neural Networks (NoNN), was proposed, which partitions the filters in the final convolutional layer of the original NN into multiple independent subsets and derives smaller NNs out of each subset. However, NoNN also has limitations in that the partitioning result may be unbalanced and it considerably compromises the correlation between filters in the original NN, which may result in an unacceptable accuracy degradation in case of communication failure. In this paper, in order to overcome these issues, we propose to enhance the partitioning strategy of NoNN in two aspects. First, we enhance the redundancy of the filters that are used to derive multiple smaller NNs by means of averaging to increase the immunity of the distributed NN to communication failure. Second, we propose a novel partitioning technique, modified from Eigenvector-based partitioning, to preserve the correlation between filters as much as possible while keeping the consistent number of filters distributed to each device. Throughout extensive experiments with the CIFAR-100 (Canadian Institute For Advanced Research-100) dataset, it has been observed that the proposed approach maintains high inference accuracy (over 70%, 1.53× improvement over the state-of-the-art approach), on average, even when a half of eight devices in a distributed NN fail to deliver their partial inference results.

scholarly journals Adversarially Robust Distillation

2020 ◽  
Vol 34 (04) ◽  
pp. 3996-4003
Author(s):  
Micah Goldblum ◽  
Liam Fowl ◽  
Soheil Feizi ◽  
Tom Goldstein
Keyword(s):  
Neural Networks ◽  
High Performance ◽  
State Of The Art ◽  
Test Accuracy ◽  
Training Methods ◽  
Knowledge Distillation ◽  
Adversarial Training ◽  
Student Models ◽  
Small Models ◽  
High Test

Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture in terms of robust accuracy, surpassing state-of-the-art methods on standard robustness benchmarks. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.

scholarly journals SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

2021 ◽  
Vol 40 (3) ◽  
pp. 1-13
Author(s):  
Lumin Yang ◽  
Jiajie Zhuang ◽  
Hongbo Fu ◽  
Xiangzhi Wei ◽  
Kun Zhou ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Large Scale ◽  
State Of The Art ◽  
Semantic Segmentation ◽  
Structure Information ◽  
Graph Neural Networks ◽  
Node Labels ◽  
Point Level

We introduce SketchGNN , a convolutional graph neural network for semantic segmentation and labeling of freehand vector sketches. We treat an input stroke-based sketch as a graph with nodes representing the sampled points along input strokes and edges encoding the stroke structure information. To predict the per-node labels, our SketchGNN uses graph convolution and a static-dynamic branching network architecture to extract the features at three levels, i.e., point-level, stroke-level, and sketch-level. SketchGNN significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentation (by 11.2% in the pixel-based metric and 18.2% in the component-based metric over a large-scale challenging SPG dataset) and has magnitudes fewer parameters than both image-based and sequence-based methods.

scholarly journals Neural network activation similarity: a new measure to assist decision making in chemical toxicology

2020 ◽  
Vol 11 (28) ◽  
pp. 7335-7348 ◽  
Author(s):  
Timothy E. H. Allen ◽  
Andrew J. Wedlake ◽  
Elena Gelžinytė ◽  
Charles Gong ◽  
Jonathan M. Goodman ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Decision Making ◽  
Deep Learning ◽  
Test Data ◽  
High Performance ◽  
Chemical Binding ◽  
Biological Targets ◽  
Network Activation ◽  
Chemical Toxicology

Deep learning neural networks, constructed for the prediction of chemical binding at 79 pharmacologically important human biological targets, show extremely high performance on test data (accuracy 92.2 ± 4.2%, MCC 0.814 ± 0.093, ROC-AUC 0.96 ± 0.04).

scholarly journals Fast and Deep Graph Neural Networks

2020 ◽  
Vol 34 (04) ◽  
pp. 3898-3905 ◽  
Author(s):  
Claudio Gallicchio ◽  
Alessio Micheli
Keyword(s):  
Neural Network ◽  
Dynamical System ◽  
Neural Networks ◽  
State Of The Art ◽  
Input Graph ◽  
Sparse Networks ◽  
Set Up ◽  
The Stability ◽  
Graph Neural Networks ◽  
Architectural Organization

We address the efficiency issue for the construction of a deep graph neural network (GNN). The approach exploits the idea of representing each input graph as a fixed point of a dynamical system (implemented through a recurrent neural network), and leverages a deep architectural organization of the recurrent units. Efficiency is gained by many aspects, including the use of small and very sparse networks, where the weights of the recurrent units are left untrained under the stability condition introduced in this work. This can be viewed as a way to study the intrinsic power of the architecture of a deep GNN, and also to provide insights for the set-up of more complex fully-trained models. Through experimental results, we show that even without training of the recurrent connections, the architecture of small deep GNN is surprisingly able to achieve or improve the state-of-the-art performance on a significant set of tasks in the field of graphs classification.

scholarly journals Optimizing Convolution Neural Network on the TI C6678 multicore DSP

2018 ◽  
Vol 246 ◽  
pp. 03044 ◽  
Author(s):  
Guozhao Zeng ◽  
Xiao Hu ◽  
Yueyue Chen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Processing ◽  
Deep Learning ◽  
Digital Signal Processor ◽  
High Performance ◽  
Digital Signal ◽  
Automatic Translation ◽  
Convolution Operation ◽  
Multicore Dsp

Convolutional Neural Networks (CNNs) have become the most advanced algorithms for deep learning. They are widely used in image processing, object detection and automatic translation. As the demand for CNNs continues to increase, the platforms on which they are deployed continue to expand. As an excellent low-power, high-performance, embedded solution, Digital Signal Processor (DSP) is used frequently in many key areas. This paper attempts to deploy the CNN to Texas Instruments (TI)’s TMS320C6678 multi-core DSP and optimize the main operations (convolution) to accommodate the DSP structure. The efficiency of the improved convolution operation has increased by tens of times.

scholarly journals Sentiment Classification Using Convolutional Neural Networks

2019 ◽  
Vol 9 (11) ◽  
pp. 2347 ◽  
Author(s):  
Hannah Kim ◽  
Young-Seob Jeong
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Text Classification ◽  
State Of The Art ◽  
Sentiment Classification ◽  
Learning Models ◽  
Text Data ◽  
Textual Data ◽  
Better Than

As the number of textual data is exponentially increasing, it becomes more important to develop models to analyze the text data automatically. The texts may contain various labels such as gender, age, country, sentiment, and so forth. Using such labels may bring benefits to some industrial fields, so many studies of text classification have appeared. Recently, the Convolutional Neural Network (CNN) has been adopted for the task of text classification and has shown quite successful results. In this paper, we propose convolutional neural networks for the task of sentiment classification. Through experiments with three well-known datasets, we show that employing consecutive convolutional layers is effective for relatively longer texts, and our networks are better than other state-of-the-art deep learning models.

scholarly journals Uncertainty-Based Human-in-the-Loop Deep Learning for Land Cover Segmentation

2020 ◽  
Vol 12 (22) ◽  
pp. 3836
Author(s):  
Carlos García Rodríguez ◽  
Jordi Vitrià ◽  
Oscar Mora
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Land Cover ◽  
Satellite Images ◽  
State Of The Art ◽  
Low Cost ◽  
Human Intervention ◽  
Human In The Loop ◽  
Learning Techniques

In recent years, different deep learning techniques were applied to segment aerial and satellite images. Nevertheless, state of the art techniques for land cover segmentation does not provide accurate results to be used in real applications. This is a problem faced by institutions and companies that want to replace time-consuming and exhausting human work with AI technology. In this work, we propose a method that combines deep learning with a human-in-the-loop strategy to achieve expert-level results at a low cost. We use a neural network to segment the images. In parallel, another network is used to measure uncertainty for predicted pixels. Finally, we combine these neural networks with a human-in-the-loop approach to produce correct predictions as if developed by human photointerpreters. Applying this methodology shows that we can increase the accuracy of land cover segmentation tasks while decreasing human intervention.

Heart Disease Classification Using Artificial Neural Networks

2015 ◽  
Vol 781 ◽  
pp. 624-627 ◽  
Author(s):  
Rati Wongsathan ◽  
Pasit Pothong
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Decision Making ◽  
Heart Disease ◽  
Classification Accuracy ◽  
High Performance ◽  
Disease Classification ◽  
Generalized Regression Neural Network ◽  
The Neural Networks ◽  
Very High

Neural Networks (NNs) has emerged as an importance tool for classification in the field of decision making. The main objective of this work is to design the structure and select the optimized parameter in the neural networks to implement the heart disease classifier. Three types of neural networks, i.e. Multi-layered Perceptron Neural Network (MLP-NN), Radial Basis Function Neural Networks (RBF-NN), and Generalized Regression Neural Network (GR-NN) have been used to test the performance of heart disease classification. The classification accuracy obtained by RBFNN gave a very high performance than MLP-NN and GR-NN respectively. The performance of accuracy is very promising compared with the previously reported another type of neural networks.

scholarly journals Investigation of optimal configurations of a convolutional neural network for the identification of objects in real-time

2019 ◽  
pp. 417-423
Author(s):  
M A Isayev ◽  
D A Savelyev
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Real Time ◽  
State Of The Art ◽  
Average Precision ◽  
The Core ◽  
Particular Solution ◽  
Optimal Configurations

The comparison of different convolutional neural networks which are the core of the most actual solutions in the computer vision area is considers in hhe paper. The study includes benchmarks of this state-of-the-art solutions by some criteria, such as mAP (mean average precision), FPS (frames per seconds), for the possibility of real-time usability. It is concluded on the best convolutional neural network model and deep learning methods that were used at particular solution.

scholarly journals Interpolation Consistency Training for Semi-supervised Learning

2019 ◽  
Author(s):  
Vikas Verma ◽  
Alex Lamb ◽  
Juho Kannala ◽  
Yoshua Bengio ◽  
David Lopez-Paz
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Supervised Learning ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Data Distribution ◽  
Network Architectures ◽  
Low Density ◽  
Decision Boundary ◽  
Classification Problems

We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. In classification problems, ICT moves the decision boundary to low-density regions of the data distribution. Our experiments show that ICT achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark dataset.

Sign in / Sign up

Export Citation Format

Share Document