scholarly journals Pruning from Scratch

2020 ◽  
Vol 34 (07) ◽  
pp. 12273-12280
Author(s):  
Yulong Wang ◽  
Xiaolu Zhang ◽  
Lingxi Xie ◽  
Jun Zhou ◽  
Hang Su ◽  
...  
Keyword(s):  
Neural Networks ◽  
Search Space ◽  
Research Field ◽  
Classification Models ◽  
Important Research ◽  
Potential Models ◽  
Computational Costs ◽  
Network Pruning ◽  
Parameterized Model ◽  
Pruning Methods

Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e.g., channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch with little training overhead. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.

Multi-granularity pruning for deep residual networks

2020 ◽  
Vol 39 (5) ◽  
pp. 7403-7410
Author(s):  
Yangke Huang ◽  
Zhiming Wang
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Compression Ratio ◽  
Gradient Descent ◽  
Computational Cost ◽  
Deep Convolutional Neural Networks ◽  
Acceleration Ratio ◽  
Network Pruning ◽  
High Computational Cost ◽  
Pruning Methods

Network pruning has been widely used to reduce the high computational cost of deep convolutional neural networks(CNNs). The dominant pruning methods, channel pruning, removes filters in layers based on their importance or sparsity training. But these methods often give limited acceleration ratio and encounter difficulties when pruning CNNs with skip connections. Block pruning methods take a sequence of consecutive layers (e.g., Conv-BN-ReLu) as a block and remove entire block each time. However, previous methods usually introduce new parameters to help pruning and lead additional parameters and extra computations. This work proposes a novel multi-granularity pruning approach that combines block pruning with channel pruning (BPCP). The block pruning (BP) module remove blocks by directly searches the redundant blocks with gradient descent and leaves no extra parameters in final models, which is friendly to hardware optimization. The channel pruning (CP) module remove redundant channels based on importance criteria and handles CNNs with skip connections properly, which further improves the overall compression ratio. As a result, for CIFAR10, BPCP reduces the number of parameters and MACs of a ResNet56 model up to 78.9% and 80.3% respectively with <3% accuracy drop. In terms of speed, it gives a 3.17 acceleration ratio. Our code has been made available at https://github.com/Pokemon-Huang/BPCP.

Identification, Prediction and Data Analysis of Noncoding RNAs: A Review

2019 ◽  
Vol 15 (3) ◽  
pp. 216-230 ◽  
Author(s):  
Abbasali Emamjomeh ◽  
Javad Zahiri ◽  
Mehrdad Asadian ◽  
Mehrdad Behmanesh ◽  
Barat A. Fakheri ◽  
...  
Keyword(s):  
Computational Methods ◽  
State Of The Art ◽  
Noncoding Rnas ◽  
Research Field ◽  
Design Strategies ◽  
Structural Prediction ◽  
Computational Costs ◽  
Cellular Processes ◽  
Laboratory Techniques ◽  
Fast Prediction

Background:Noncoding RNAs (ncRNAs) which play an important role in various cellular processes are important in medicine as well as in drug design strategies. Different studies have shown that ncRNAs are dis-regulated in cancer cells and play an important role in human tumorigenesis. Therefore, it is important to identify and predict such molecules by experimental and computational methods, respectively. However, to avoid expensive experimental methods, computational algorithms have been developed for accurately and fast prediction of ncRNAs.Objective:The aim of this review was to introduce the experimental and computational methods to identify and predict ncRNAs structure. Also, we explained the ncRNA’s roles in cellular processes and drugs design, briefly.Method:In this survey, we will introduce ncRNAs and their roles in biological and medicinal processes. Then, some important laboratory techniques will be studied to identify ncRNAs. Finally, the state-of-the-art models and algorithms will be introduced along with important tools and databases.Results:The results showed that the integration of experimental and computational approaches improves to identify ncRNAs. Moreover, the high accurate databases, algorithms and tools were compared to predict the ncRNAs.Conclusion:ncRNAs prediction is an exciting research field, but there are different difficulties. It requires accurate and reliable algorithms and tools. Also, it should be mentioned that computational costs of such algorithm including running time and usage memory are very important. Finally, some suggestions were presented to improve computational methods of ncRNAs gene and structural prediction.

scholarly journals Very Fast and Accurate Procedure for the Characterization of Photovoltaic Panels from Datasheet Information

2014 ◽  
Vol 2014 ◽  
pp. 1-10 ◽  
Author(s):  
Antonino Laudani ◽  
Francesco Riganti Fulginei ◽  
Alessandro Salvini ◽  
Gabriele Maria Lozito ◽  
Salvatore Coco
Keyword(s):  
Numerical Methods ◽  
Search Space ◽  
Extraction Process ◽  
Reduced Form ◽  
Photovoltaic Panels ◽  
Approximation Techniques ◽  
Computational Costs ◽  
Minimization Technique ◽  
Accurate Procedure

In recent years several numerical methods have been proposed to identify the five-parameter model of photovoltaic panels from manufacturer datasheets also by introducing simplification or approximation techniques. In this paper we present a fast and accurate procedure for obtaining the parameters of the five-parameter model by starting from its reduced form. The procedure allows characterizing, in few seconds, thousands of photovoltaic panels present on the standard databases. It introduces and takes advantage of further important mathematical considerations without any model simplifications or data approximations. In particular the five parameters are divided in two groups, independent and dependent parameters, in order to reduce the dimensions of the search space. The partitioning of the parameters provides a strong advantage in terms of convergence, computational costs, and execution time of the present approach. Validations on thousands of photovoltaic panels are presented that show how it is possible to make easy and efficient the extraction process of the five parameters, without taking care of choosing a specific solver algorithm but simply by using any deterministic optimization/minimization technique.

scholarly journals An Adversarial Generative Network for Crop Classification from Remote Sensing Timeseries Images

2020 ◽  
Vol 13 (1) ◽  
pp. 65
Author(s):  
Jingtao Li ◽  
Yonglin Shen ◽  
Chao Yang
Keyword(s):  
Remote Sensing ◽  
Neural Networks ◽  
Short Term Memory ◽  
Complete Classification ◽  
Support Vector ◽  
Classification Models ◽  
Training Samples ◽  
Agricultural Applications ◽  
Crop Classification ◽  
Increasing Demand

Due to the increasing demand for the monitoring of crop conditions and food production, it is a challenging and meaningful task to identify crops from remote sensing images. The state-of the-art crop classification models are mostly built on supervised classification models such as support vector machines (SVM), convolutional neural networks (CNN), and long- and short-term memory neural networks (LSTM). Meanwhile, as an unsupervised generative model, the adversarial generative network (GAN) is rarely used to complete classification tasks for agricultural applications. In this work, we propose a new method that combines GAN, CNN, and LSTM models to classify crops of corn and soybeans from remote sensing time-series images, in which GAN’s discriminator was used as the final classifier. The method is feasible on the condition that the training samples are small, and it fully takes advantage of spectral, spatial, and phenology features of crops from satellite data. The classification experiments were conducted on crops of corn, soybeans, and others. To verify the effectiveness of the proposed method, comparisons with models of SVM, SegNet, CNN, LSTM, and different combinations were also conducted. The results show that our method achieved the best classification results, with the Kappa coefficient of 0.7933 and overall accuracy of 0.86. Experiments in other study areas also demonstrate the extensibility of the proposed method.

scholarly journals Classification of Approximal Caries in Bitewing Radiographs Using Convolutional Neural Networks

Sensors ◽  
2021 ◽  
Vol 21 (15) ◽  
pp. 5192
Author(s):  
Maira Moran ◽  
Marcelo Faria ◽  
Gilson Giraldi ◽  
Luciana Bastos ◽  
Larissa Oliveira ◽  
...  
Keyword(s):  
Neural Networks ◽  
Dental Caries ◽  
Convolutional Neural Networks ◽  
Data Augmentation ◽  
Clinical Analysis ◽  
Diagnostic Process ◽  
Radiographic Evaluation ◽  
Classification Models ◽  
Approximal Caries ◽  
Lesion Severity

Dental caries is an extremely common problem in dentistry that affects a significant part of the population. Approximal caries are especially difficult to identify because their position makes clinical analysis difficult. Radiographic evaluation—more specifically, bitewing images—are mostly used in such cases. However, incorrect interpretations may interfere with the diagnostic process. To aid dentists in caries evaluation, computational methods and tools can be used. In this work, we propose a new method that combines image processing techniques and convolutional neural networks to identify approximal dental caries in bitewing radiographic images and classify them according to lesion severity. For this study, we acquired 112 bitewing radiographs. From these exams, we extracted individual tooth images from each exam, applied a data augmentation process, and used the resulting images to train CNN classification models. The tooth images were previously labeled by experts to denote the defined classes. We evaluated classification models based on the Inception and ResNet architectures using three different learning rates: 0.1, 0.01, and 0.001. The training process included 2000 iterations, and the best results were achieved by the Inception model with a 0.001 learning rate, whose accuracy on the test set was 73.3%. The results can be considered promising and suggest that the proposed method could be used to assist dentists in the evaluation of bitewing images, and the definition of lesion severity and appropriate treatments.

scholarly journals Generalization-Based Acquisition of Training Data for Motor Primitive Learning by Neural Networks

2021 ◽  
Vol 11 (3) ◽  
pp. 1013
Author(s):  
Zvezdan Lončarević ◽  
Rok Pahič ◽  
Aleš Ude ◽  
Andrej Gams
Keyword(s):  
Neural Networks ◽  
Dimensionality Reduction ◽  
Gaussian Process Regression ◽  
Search Space ◽  
Robot Learning ◽  
Training Data ◽  
Practical Applications ◽  
Latent Space ◽  
Real Robot ◽  
Low Dimensional

Autonomous robot learning in unstructured environments often faces the problem that the dimensionality of the search space is too large for practical applications. Dimensionality reduction techniques have been developed to address this problem and describe motor skills in low-dimensional latent spaces. Most of these techniques require the availability of a sufficiently large database of example task executions to compute the latent space. However, the generation of many example task executions on a real robot is tedious, and prone to errors and equipment failures. The main result of this paper is a new approach for efficient database gathering by performing a small number of task executions with a real robot and applying statistical generalization, e.g., Gaussian process regression, to generate more data. We have shown in our experiments that the data generated this way can be used for dimensionality reduction with autoencoder neural networks. The resulting latent spaces can be exploited to implement robot learning more efficiently. The proposed approach has been evaluated on the problem of robotic throwing at a target. Simulation and real-world results with a humanoid robot TALOS are provided. They confirm the effectiveness of generalization-based database acquisition and the efficiency of learning in a low-dimensional latent space.

scholarly journals A Green Prospective for Learned Post-processing in Sparse-view Tomographic Reconstruction

2021 ◽  
Author(s):  
Elena Morotti ◽  
Davide Evangelista ◽  
Elena Loli Piccolomini
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Tomographic Reconstruction ◽  
Research Field ◽  
Filtered Backprojection ◽  
Post Processing ◽  
Training Set ◽  
Imaging Reconstruction ◽  
Active Research

Deep Learning is developing interesting tools which are of great interest for inverse imaging applications. In this work, we consider a medical imaging reconstruction task from subsampled measurements, which is an active research field where Convolutional Neural Networks have already revealed their great potential. However, the commonly used architectures are very deep and, hence, prone to overfitting and unfeasible for clinical usages. Inspired by the ideas of the green-AI literature, we here propose a shallow neural network to perform an efficient Learned Post-Processing on images roughly reconstructed by the filtered backprojection algorithm. The results obtained on images from the training set and on unseen images, using both the non-expensive network and the widely used very deep ResUNet show that the proposed network computes images of comparable or higher quality in about one fourth of time.

scholarly journals Beyond Network Pruning: a Joint Search-and-Training Approach

2020 ◽  
Author(s):  
Xiaotong Lu ◽  
Han Huang ◽  
Weisheng Dong ◽  
Xin Li ◽  
Guangming Shi
Keyword(s):  
Random Perturbation ◽  
Search Space ◽  
Fine Tuning ◽  
Superior Performance ◽  
Network Pruning ◽  
Training Approach ◽  
Target Network ◽  
Tuning Strategy ◽  
Coarse To Fine ◽  
And Training

Network pruning has been proposed as a remedy for alleviating the over-parameterization problem of deep neural networks. However, its value has been recently challenged especially from the perspective of neural architecture search (NAS). We challenge the conventional wisdom of pruning-after-training by proposing a joint search-and-training approach that directly learns a compact network from the scratch. By treating pruning as a search strategy, we present two new insights in this paper: 1) it is possible to expand the search space of networking pruning by associating each filter with a learnable weight; 2) joint search-and-training can be conducted iteratively to maximize the learning efficiency. More specifically, we propose a coarse-to-fine tuning strategy to iteratively sample and update compact sub-network to approximate the target network. The weights associated with network filters will be accordingly updated by joint search-and-training to reflect learned knowledge in NAS space. Moreover, we introduce strategies of random perturbation (inspired by Monte Carlo) and flexible thresholding (inspired by Reinforcement Learning) to adjust the weight and size of each layer. Extensive experiments on ResNet and VGGNet demonstrate the superior performance of our proposed method on popular datasets including CIFAR10, CIFAR100 and ImageNet.

The Genetic Algorithm

2019 ◽  
pp. 154-178
Author(s):  
Burcu Adıguzel Mercangöz ◽  
Ergun Eroglu
Keyword(s):  
Genetic Algorithm ◽  
Mathematical Models ◽  
Portfolio Optimization ◽  
Optimization Problem ◽  
Optimization Problems ◽  
Biological Evolution ◽  
Research Field ◽  
Risk Level ◽  
Important Research ◽  
Portfolio Optimization Problem

The portfolio optimization is an important research field of the financial sciences. In portfolio optimization problems, it is aimed to create portfolios by giving the best return at a certain risk level from the asset pool or by selecting assets that give the lowest risk at a certain level of return. The diversity of the portfolio gives opportunity to increase the return by minimizing the risk. As a powerful alternative to the mathematical models, heuristics is used widely to solve the portfolio optimization problems. The genetic algorithm (GA) is a technique that is inspired by the biological evolution. While this book considers the heuristics methods for the portfolio optimization problems, this chapter will give the implementing steps of the GA clearly and apply this method to a portfolio optimization problem in a basic example.

Sign in / Sign up

Export Citation Format

Share Document