scholarly journals Smooth Group L1/2 Regularization for Pruning Convolutional Neural Networks

Symmetry ◽  
2022 ◽  
Vol 14 (1) ◽  
pp. 154
Author(s):  
Yuan Bao ◽  
Zhaobin Liu ◽  
Zhongxuan Luo ◽  
Sibo Yang
Keyword(s):  
Neural Networks ◽  
Regularization Method ◽  
Classification Performance ◽  
Feasibility Analysis ◽  
Group Level ◽  
Hidden Node ◽  
Convolution Filter ◽  
Fully Connected ◽  
Hidden Nodes ◽  
Selection Of

In this paper, a novel smooth group L1/2 (SGL1/2) regularization method is proposed for pruning hidden nodes of the fully connected layer in convolution neural networks. Usually, the selection of nodes and weights is based on experience, and the convolution filter is symmetric in the convolution neural network. The main contribution of SGL1/2 is to try to approximate the weights to 0 at the group level. Therefore, we will be able to prune the hidden node if the corresponding weights are all close to 0. Furthermore, the feasibility analysis of this new method is carried out under some reasonable assumptions due to the smooth function. The numerical results demonstrate the superiority of the SGL1/2 method with respect to sparsity, without damaging the classification performance.

scholarly journals Can Pre-Trained Convolutional Neural Networks be directly used as a Feature Extractor for Video-based Neonatal Sleep and Wake Classification?

2020 ◽  
Author(s):  
Muhammad Awais ◽  
Xi Long ◽  
Bin Yin ◽  
Chen chen ◽  
Saeed Akbarzadeh ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Video Recording ◽  
Principal Component ◽  
Classification Performance ◽  
Support Vector ◽  
Video Frames ◽  
Feature Extractor ◽  
Fully Connected

Abstract Objective: In this paper, we propose to evaluate the use of a pre-trained convolutional neural networks (CNNs) as a features extractor followed by the Principal Component Analysis (PCA) to find the best discriminant features to perform classification using support vector machine (SVM) algorithm for neonatal sleep and wake states using Fluke® facial video frames. Using pre-trained CNNs as feature extractor would hugely reduce the effort of collecting new neonatal data for training a neural network which could be computationally very expensive. The features are extracted after fully connected layers (FCL’s), where we compare several pre-trained CNNs, e.g., VGG16, VGG19, InceptionV3, GoogLeNet, ResNet, and AlexNet. Results: From around 2-h Fluke® video recording of seven neonate, we achieved a modest classification performance with an accuracy, sensitivity, and specificity of 65.3%, 69.8%, 61.0%, respectively with AlexNet using Fluke® (RGB) video frames. This indicates that using a pre-trained model as a feature extractor could not fully suffice for highly reliable sleep and wake classification in neonates. Therefore, in future a dedicated neural network trained on neonatal data or a transfer learning approach is required.

scholarly journals Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks

2020 ◽  
Vol 34 (04) ◽  
pp. 4394-4403
Author(s):  
Sekitoshi Kanai ◽  
Yasutoshi Ida ◽  
Yasuhiro Fujiwara ◽  
Masanori Yamada ◽  
Shuichi Adachi
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Regularization Method ◽  
Regularization Methods ◽  
Frequency Noise ◽  
Weight Decay ◽  
Structural Sensitivity ◽  
Convolution Filter ◽  
Gradient Based ◽  
Standard Regularization

We propose Absum, which is a regularization method for improving adversarial robustness of convolutional neural networks (CNNs). Although CNNs can accurately recognize images, recent studies have shown that the convolution operations in CNNs commonly have structural sensitivity to specific noise composed of Fourier basis functions. By exploiting this sensitivity, they proposed a simple black-box adversarial attack: Single Fourier attack. To reduce structural sensitivity, we can use regularization of convolution filter weights since the sensitivity of linear transform can be assessed by the norm of the weights. However, standard regularization methods can prevent minimization of the loss function because they impose a tight constraint for obtaining high robustness. To solve this problem, Absum imposes a loose constraint; it penalizes the absolute values of the summation of the parameters in the convolution layers. Absum can improve robustness against single Fourier attack while being as simple and efficient as standard regularization methods (e.g., weight decay and L1 regularization). Our experiments demonstrate that Absum improves robustness against single Fourier attack more than standard regularization methods. Furthermore, we reveal that robust CNNs with Absum are more robust against transferred attacks due to decreasing the common sensitivity and against high-frequency noise than standard regularization methods. We also reveal that Absum can improve robustness against gradient-based attacks (projected gradient descent) when used with adversarial training.

scholarly journals Can Pre-Trained Convolutional Neural Networks be used as Feature Extractors for Video-based Neonatal Sleep and Wake Classification?

2020 ◽  
Author(s):  
Muhammad Awais ◽  
Xi Long ◽  
Bin Yin ◽  
Chen chen ◽  
Saeed Akbarzadeh ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Video Recording ◽  
Principal Component ◽  
Classification Performance ◽  
Support Vector ◽  
Video Frames ◽  
Feature Extractor ◽  
Fully Connected

Abstract Objective In this paper, we propose to evaluate the use of a pre-trained convolutional neural networks (CNN’s) as a features extractor followed by the Principal Component Analysis (PCA) to find the best discriminant features to perform classification using support vector machine (SVM) algorithm for neonatal sleep and wake states using Fluke® facial video frames. Using pre-trained CNN’s as feature extractor would hugely reduce the effort of collecting new neonatal data for training a neural network which could be computationally very expensive. The features are extracted after fully connected layers (FCL’s), where we compare several pre-trained CNN’s, e.g., VGG16, VGG19, InceptionV3, GoogLeNet, ResNet, and AlexNet. Results From around 2-h Fluke® video recording of seven neonate, we achieved a modest classification performance with an accuracy, sensitivity, and specificity of 65.3%, 69.8%, 61.0%, respectively with AlexNet using Fluke® (RGB) video frames. This indicates that using a pre-trained model as a feature extractor could not fully suffice for highly reliable sleep and wake classification in neonates. Therefore, in future a dedicated neural network trained on neonatal data or a transfer learning approach is required.

scholarly journals Can Pre-Trained Convolutional Neural Networks be used as Feature Extractors for Video-based Neonatal Sleep and Wake Classification?

2020 ◽  
Author(s):  
Muhammad Awais ◽  
Xi Long ◽  
Bin Yin ◽  
Chen chen ◽  
Saeed Akbarzadeh ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Video Recording ◽  
Principal Component ◽  
Classification Performance ◽  
Support Vector ◽  
Video Frames ◽  
Feature Extractor ◽  
Fully Connected

Abstract Objective: In this paper, we propose to evaluate the use of a pre-trained convolutional neural networks (CNNs) as a features extractor followed by the Principal Component Analysis (PCA) to find the best discriminant features to perform classification using support vector machine (SVM) algorithm for neonatal sleep and wake states using Fluke® facial video frames. Using pre-trained CNNs as feature extractor would hugely reduce the effort of collecting new neonatal data for training a neural network which could be computationally very expensive. The features are extracted after fully connected layers (FCL’s), where we compare several pre-trained CNNs, e.g., VGG16, VGG19, InceptionV3, GoogLeNet, ResNet, and AlexNet.Results: From around 2-h Fluke® video recording of seven neonate, we achieved a modest classification performance with an accuracy, sensitivity, and specificity of 65.3%, 69.8%, 61.0%, respectively with AlexNet using Fluke® (RGB) video frames. This indicates that using a pre-trained model as a feature extractor could not fully suffice for highly reliable sleep and wake classification in neonates. Therefore, in future a dedicated neural network trained on neonatal data or a transfer learning approach is required.

A novel structured sparse fully connected layer in convolutional neural networks

2021 ◽  
Author(s):  
Naoki Matsumura ◽  
Yasuaki Ito ◽  
Koji Nakano ◽  
Akihiko Kasagi ◽  
Tsuguchika Tabaru
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Fully Connected

scholarly journals Structural Damage Identification of Composite Rotors Based on Fully Connected Neural Networks and Convolutional Neural Networks

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 2005
Author(s):  
Veronika Scholz ◽  
Peter Winkler ◽  
Andreas Hornig ◽  
Maik Gude ◽  
Angelos Filippatos
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Composite Structures ◽  
Damage Identification ◽  
Vibration Response ◽  
Damage Initiation ◽  
Matrix Cracks ◽  
Operational Life ◽  
Multiple Data Sets ◽  
Fully Connected

Damage identification of composite structures is a major ongoing challenge for a secure operational life-cycle due to the complex, gradual damage behaviour of composite materials. Especially for composite rotors in aero-engines and wind-turbines, a cost-intensive maintenance service has to be performed in order to avoid critical failure. A major advantage of composite structures is that they are able to safely operate after damage initiation and under ongoing damage propagation. Therefore, a robust, efficient diagnostic damage identification method would allow monitoring the damage process with intervention occurring only when necessary. This study investigates the structural vibration response of composite rotors by applying machine learning methods and the ability to identify, localise and quantify the present damage. To this end, multiple fully connected neural networks and convolutional neural networks were trained on vibration response spectra from damaged composite rotors with barely visible damage, mostly matrix cracks and local delaminations using dimensionality reduction and data augmentation. A databank containing 720 simulated test cases with different damage states is used as a basis for the generation of multiple data sets. The trained models are tested using k-fold cross validation and they are evaluated based on the sensitivity, specificity and accuracy. Convolutional neural networks perform slightly better providing a performance accuracy of up to 99.3% for the damage localisation and quantification.

scholarly journals A New Hyper-Parameter Optimization Method for Power Load Forecast Based on Recurrent Neural Networks

Algorithms ◽  
2021 ◽  
Vol 14 (6) ◽  
pp. 163
Author(s):  
Yaru Li ◽  
Yulai Zhang ◽  
Yongping Cai
Keyword(s):  
Neural Networks ◽  
Parameter Optimization ◽  
Recurrent Neural Networks ◽  
Critical Role ◽  
Optimization Method ◽  
Bayesian Optimization ◽  
Power Load ◽  
Gradient Calculation ◽  
Truncated Newton ◽  
Selection Of

The selection of the hyper-parameters plays a critical role in the task of prediction based on the recurrent neural networks (RNN). Traditionally, the hyper-parameters of the machine learning models are selected by simulations as well as human experiences. In recent years, multiple algorithms based on Bayesian optimization (BO) are developed to determine the optimal values of the hyper-parameters. In most of these methods, gradients are required to be calculated. In this work, the particle swarm optimization (PSO) is used under the BO framework to develop a new method for hyper-parameter optimization. The proposed algorithm (BO-PSO) is free of gradient calculation and the particles can be optimized in parallel naturally. So the computational complexity can be effectively reduced which means better hyper-parameters can be obtained under the same amount of calculation. Experiments are done on real world power load data,where the proposed method outperforms the existing state-of-the-art algorithms,BO with limit-BFGS-bound (BO-L-BFGS-B) and BO with truncated-newton (BO-TNC),in terms of the prediction accuracy. The errors of the prediction result in different models show that BO-PSO is an effective hyper-parameter optimization method.

Sign in / Sign up

Export Citation Format

Share Document