scholarly journals Deterministic Binary Filters for Convolutional Neural Networks

2018 ◽  
Author(s):  
Vincent W.-S. Tseng ◽  
Sourav Bhattacharya ◽  
Javier Fernández Marqués ◽  
Milad Alizadeh ◽  
Catherine Tong ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Filter Design ◽  
Size Reduction ◽  
Floating Point ◽  
Network Architectures ◽  
Comparable Accuracy ◽  
Model Size ◽  
Weighting Coefficients ◽  
Binary Networks

We propose Deterministic Binary Filters, an approach to Convolutional Neural Networks that learns weighting coefficients of predefined orthogonal binary basis instead of the conventional approach of learning directly the convolutional filters. This approach results in model architectures with significantly fewer parameters (4x to 16x) and smaller model sizes (32x due to the use of binary rather than floating point precision). We show our deterministic filter design can be integrated into well-known network architectures (such as ResNet and SqueezeNet) with as little as 2% loss of accuracy (under datasets like CIFAR-10). Under ImageNet, they result in 3x model size reduction compared to sub-megabyte binary networks while reaching comparable accuracy levels.

scholarly journals Layer-Wise Compressive Training for Convolutional Neural Networks

2018 ◽  
Vol 11 (1) ◽  
pp. 7 ◽  
Author(s):  
Matteo Grimaldi ◽  
Valerio Tenace ◽  
Andrea Calimera
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Gradient Descent ◽  
Computational Models ◽  
Stochastic Gradient Descent ◽  
Training Algorithm ◽  
Heuristic Rules ◽  
Human Capabilities ◽  
Model Size ◽  
Large Model

Convolutional Neural Networks (CNNs) are brain-inspired computational models designed to recognize patterns. Recent advances demonstrate that CNNs are able to achieve, and often exceed, human capabilities in many application domains. Made of several millions of parameters, even the simplest CNN shows large model size. This characteristic is a serious concern for the deployment on resource-constrained embedded-systems, where compression stages are needed to meet the stringent hardware constraints. In this paper, we introduce a novel accuracy-driven compressive training algorithm. It consists of a two-stage flow: first, layers are sorted by means of heuristic rules according to their significance; second, a modified stochastic gradient descent optimization is applied on less significant layers such that their representation is collapsed into a constrained subspace. Experimental results demonstrate that our approach achieves remarkable compression rates with low accuracy loss (<1%).

scholarly journals Implementation of Lightweight Convolutional Neural Networks via Layer-Wise Differentiable Compression

Sensors ◽  
2021 ◽  
Vol 21 (10) ◽  
pp. 3464
Author(s):  
Huabin Diao ◽  
Yuexing Hao ◽  
Shaoyun Xu ◽  
Gongyan Li
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
Compression Rate ◽  
Software Support ◽  
Model Size ◽  
Significant Compression ◽  
Selection Operator ◽  
The Impact ◽  
Constrained Devices

Convolutional neural networks (CNNs) have achieved significant breakthroughs in various domains, such as natural language processing (NLP), and computer vision. However, performance improvement is often accompanied by large model size and computation costs, which make it not suitable for resource-constrained devices. Consequently, there is an urgent need to compress CNNs, so as to reduce model size and computation costs. This paper proposes a layer-wise differentiable compression (LWDC) algorithm for compressing CNNs structurally. A differentiable selection operator OS is embedded in the model to compress and train the model simultaneously by gradient descent in one go. Instead of pruning parameters from redundant operators by contrast to most of the existing methods, our method replaces the original bulky operators with more lightweight ones directly, which only needs to specify the set of lightweight operators and the regularization factor in advance, rather than the compression rate for each layer. The compressed model produced by our method is generic and does not need any special hardware/software support. Experimental results on CIFAR-10, CIFAR-100 and ImageNet have demonstrated the effectiveness of our method. LWDC obtains more significant compression than state-of-the-art methods in most cases, while having lower performance degradation. The impact of lightweight operators and regularization factor on the compression rate and accuracy also is evaluated.

scholarly journals Improvement for Convolutional Neural Networks in Image Classification Using Long Skip Connection

2021 ◽  
Vol 11 (5) ◽  
pp. 2092
Author(s):  
Hong Hai Hoang ◽  
Hoang Hieu Trinh
Keyword(s):  
Neural Networks ◽  
Image Classification ◽  
Convolutional Neural Networks ◽  
Surface Defect ◽  
Spatial Knowledge ◽  
Floating Point ◽  
Defect Classification

In this paper, we examine and research the effect of long skip connection on convolutional neural networks (CNNs) for the tasks of image (surface defect) classification. The standard popular models only apply short skip connection inside blocks (layers with the same size). We apply the long version of residual connection on several proposed models, which aims to reuse the lost spatial knowledge from the layers close to input. For some models, Depthwise Separable Convolution is used rather than traditional convolution in order to reduce both count of parameters and floating-point operations per second (FLOPs). Comparative experiments of the newly upgraded models and some popular models have been carried out on different datasets including Bamboo strips datasets and a reduced version of ImageNet. The modified version of DenseNet 121 (we call MDenseNet 121) achieves higher validation accuracy while it has about 75% of weights and FLOPs in comparison to the original DenseNet 121.

Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection

2017 ◽  
Vol 17 (5) ◽  
pp. 1110-1128 ◽  
Author(s):  
Deegan J Atha ◽  
Mohammad R Jahanshahi
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
State Of The Art ◽  
Inspection Time ◽  
Computational Time ◽  
Network Architectures ◽  
Corrosion Detection ◽  
Neural Network Architectures

Corrosion is a major defect in structural systems that has a significant economic impact and can pose safety risks if left untended. Currently, an inspector visually assesses the condition of a structure to identify corrosion. This approach is time-consuming, tedious, and subjective. Robotic systems, such as unmanned aerial vehicles, paired with computer vision algorithms have the potential to perform autonomous damage detection that can significantly decrease inspection time and lead to more frequent and objective inspections. This study evaluates the use of convolutional neural networks for corrosion detection. A convolutional neural network learns the appropriate classification features that in traditional algorithms were hand-engineered. Eliminating the need for dependence on prior knowledge and human effort in designing features is a major advantage of convolutional neural networks. This article presents different convolutional neural network–based approaches for corrosion assessment on metallic surfaces. The effect of different color spaces, sliding window sizes, and convolutional neural network architectures are discussed. To this end, the performance of two pretrained state-of-the-art convolutional neural network architectures as well as two proposed convolutional neural network architectures are evaluated, and it is shown that convolutional neural networks outperform state-of-the-art vision-based corrosion detection approaches that are developed based on texture and color analysis using a simple multilayered perceptron network. Furthermore, it is shown that one of the proposed convolutional neural networks significantly improves the computational time in contrast with state-of-the-art pretrained convolutional neural networks while maintaining comparable performance for corrosion detection.

scholarly journals FRD-CNN: Object detection based on small-scale convolutional neural networks and feature reuse

2019 ◽  
Vol 9 (1) ◽  
Author(s):  
Wei Li ◽  
Kai Liu ◽  
Lin Yan ◽  
Fei Cheng ◽  
YunQiu Lv ◽  
...  
Keyword(s):  
Neural Networks ◽  
Object Detection ◽  
Convolutional Neural Networks ◽  
Network Performance ◽  
Thin Layers ◽  
Detection Methods ◽  
Small Scale ◽  
Pascal Voc ◽  
Model Size

Abstract Most of the recent successful object detection methods have been based on convolutional neural networks (CNNs). From previous studies, we learned that many feature reuse methods improve the network performance, but they increase the number of parameters. DenseNet uses thin layers that have fewer channels to alleviate the increase in parameters. This motivated us to find other methods for solving the increase in model size problems introduced by feature reuse methods. In this work, we employ different feature reuse methods on fire units and mobile units. We solved the problem and constructed two novel neural networks, fire-FRD-CNN and mobile-FRD-CNN. We conducted experiments with the proposed neural networks on KITTI and PASCAL VOC datasets.

scholarly journals Optimization of FireNet for Liver Lesion Classification

Electronics ◽  
2020 ◽  
Vol 9 (8) ◽  
pp. 1237
Author(s):  
Gedeon Kashala Kabe ◽  
Yuqing Song ◽  
Zhe Liu
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Liver Lesion ◽  
Superior Performance ◽  
Visual Object ◽  
Visual Object Recognition ◽  
Residual Function ◽  
Learning Techniques ◽  
Model Size

In recent years, deep learning techniques, and in particular convolutional neural networks (CNNs) methods have demonstrated a superior performance in image classification and visual object recognition. In this work, we propose a classification of four types of liver lesions, namely, hepatocellular carcinoma, metastases, hemangiomas, and healthy tissues using convolutional neural networks with a succinct model called FireNet. We improved speed for quick classification and decreased the model size and the number of parameters by using fire modules from SqueezeNet. We have used bypass connection by adding it around Fire modules for learning a residual function between input and output, and to solve the vanishing gradient problem. We have proposed a new Particle Swarm Optimization (NPSO) to optimize the network parameters in order to further boost the performance of the proposed FireNet. The experimental results show that the parameters of FireNet are 9.5 times smaller than GoogLeNet, 51.6 times smaller than AlexNet, and 75.8 smaller than ResNet. The size of FireNet is reduced 16.6 times smaller than GoogLeNet, 75 times smaller than AlexNet and 76.6 times smaller than ResNet. The final accuracy of our proposed FireNet model was 89.2%.

scholarly journals Compact and Accurate Scene Text Detector

2020 ◽  
Vol 10 (6) ◽  
pp. 2096 ◽  
Author(s):  
Minjun Jeon ◽  
Young-Seob Jeong
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Text Detection ◽  
Floating Point ◽  
Learning Models ◽  
Proposed Model ◽  
Scene Text Detection ◽  
Scene Text

Scene text detection is the task of detecting word boxes in given images. The accuracy of text detection has been greatly elevated using deep learning models, especially convolutional neural networks. Previous studies commonly aimed at developing more accurate models, but their models became computationally heavy and worse in efficiency. In this paper, we propose a new efficient model for text detection. The proposed model, namely Compact and Accurate Scene Text detector (CAST), consists of MobileNetV2 as a backbone and balanced decoder. Unlike previous studies that used standard convolutional layers as a decoder, we carefully design a balanced decoder. Through experiments with three well-known datasets, we then demonstrated that the balanced decoder and the proposed CAST are efficient and effective. The CAST was about 1.1x worse in terms of the F1 score, but 30∼115x better in terms of floating-point operations per second (FLOPS).

scholarly journals Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks

2020 ◽  
Vol 34 (04) ◽  
pp. 5355-5362
Author(s):  
Fabio Pardo ◽  
Vitaly Levdik ◽  
Petar Kormushev
Keyword(s):  
Neural Networks ◽  
Reinforcement Learning ◽  
Convolutional Neural Networks ◽  
Network Architectures ◽  
Convolutional Network ◽  
Agent Learning ◽  
Q Values

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in ε-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.

scholarly journals Process Monitoring in Friction Stir Welding Using Convolutional Neural Networks

Metals ◽  
2021 ◽  
Vol 11 (4) ◽  
pp. 535
Author(s):  
Roman Hartl ◽  
Andreas Bachmann ◽  
Jan Bernd Habedank ◽  
Thomas Semm ◽  
Michael F. Zaeh
Keyword(s):  
Neural Networks ◽  
Friction Stir Welding ◽  
Convolutional Neural Networks ◽  
Process Monitoring ◽  
Surface Defects ◽  
Network Architectures ◽  
Process Data ◽  
Friction Stir ◽  
Measured Process

Preliminary studies have shown the superiority of convolutional neural networks (CNNs) compared to other network architectures for determining the surface quality of friction stir welds. In this paper, CNNs were employed to detect cavities inside friction stir welds by evaluating inline measured process data. The aim was to determine whether CNNs are suitable for identifying surface defects exclusively, or if the approach is transferable to internal weld defects. For this purpose, 120 welds were produced and examined by ultrasonic testing, which was the basis for labeling the data as “good” or “defective.” Different types of artificial neural network were tested for predicting the placement of the welds into the defined classes. It was found that the way of labeling the data is significant for the accuracy achievable. When the complete welds were uniformly labeled as “good” or “defective,” an accuracy of 98.5% was achieved by a CNN, which was a significant improvement compared to the state of the art. When the welds were labeled segment-wise, an accuracy of 79.2% was obtained by using a CNN, showing that a segment-wise prediction of the cavities is also possible. The results confirm that CNNs are well suited for process monitoring in friction stir welding and their application enables the identification of various defect types.

Sign in / Sign up

Export Citation Format

Share Document