Greedy search algorithm for partial quantization of convolutional neural networks inspired by submodular optimization

2022 ◽  
Author(s):  
Satoki Tsuji ◽  
Fuyuka Yamada ◽  
Hiroshi Kawaguchi ◽  
Atsuki Inoue ◽  
Yasufumi Sakai
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Search Algorithm ◽  
Greedy Search ◽  
Submodular Optimization ◽  
Partial Quantization

scholarly journals Language Semantics Interpretation with an Interaction-Based Recurrent Neural Network

2021 ◽  
Vol 3 (4) ◽  
pp. 922-945
Author(s):  
Shaw-Hwa Lo ◽  
Yiqiao Yin
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Language Processing ◽  
Text Classification ◽  
Search Algorithm ◽  
Greedy Search ◽  
Text Documents ◽  
Engineering Technique ◽  
Language Semantics ◽  
Sequential Models

Text classification is a fundamental language task in Natural Language Processing. A variety of sequential models are capable of making good predictions, yet there is a lack of connection between language semantics and prediction results. This paper proposes a novel influence score (I-score), a greedy search algorithm, called Backward Dropping Algorithm (BDA), and a novel feature engineering technique called the “dagger technique”. First, the paper proposes to use the novel influence score (I-score) to detect and search for the important language semantics in text documents that are useful for making good predictions in text classification tasks. Next, a greedy search algorithm, called the Backward Dropping Algorithm, is proposed to handle long-term dependencies in the dataset. Moreover, the paper proposes a novel engineering technique called the “dagger technique” that fully preserves the relationship between the explanatory variable and the response variable. The proposed techniques can be further generalized into any feed-forward Artificial Neural Networks (ANNs) and Convolutional Neural Networks (CNNs), and any neural network. A real-world application on the Internet Movie Database (IMDB) is used and the proposed methods are applied to improve prediction performance with an 81% error reduction compared to other popular peers if I-score and “dagger technique” are not implemented.

Traditional Classifiers vs. Deep Learning for Cyberbullying Detection

2018 ◽  
pp. 133-149
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Search Algorithm ◽  
Learning Approach ◽  
Brute Force ◽  
Brute Force Search ◽  
Novel Method ◽  
Objective View ◽  
Cyberbullying Detection

In this chapter, the authors present their approach to cyberbullying detection with the use of various traditional classifiers, including a deep learning approach. Research has tackled the problem of cyberbullying detection during recent years. However, due to complexity of language used in cyberbullying, the results obtained with traditional classifiers has remained only mildly satisfying. In this chapter, the authors apply a number of traditional classifiers, used also in previous research, to obtain an objective view on to what extent each of them is suitable to the task. They also propose a novel method to automatic cyberbullying detection based on convolutional neural networks and increased feature density. The experiments performed on actual cyberbullying data showed a major advantage of the presented approach to all previous methods, including the two best performing methods so far based on SO-PMI-IR and brute-force search algorithm, presented in previous two chapters.

scholarly journals Designing convolutional neural networks with constrained evolutionary piecemeal training

2021 ◽  
Author(s):  
Dolly Sapra ◽  
Andy D. Pimentel
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Classification ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
Network Architecture ◽  
High Performance ◽  
Search Algorithm ◽  
Search Space ◽  
Pareto Optimal Set

AbstractThe automated architecture search methodology for neural networks is known as Neural Architecture Search (NAS). In recent times, Convolutional Neural Networks (CNNs) designed through NAS methodologies have achieved very high performance in several fields, for instance image classification and natural language processing. Our work is in the same domain of NAS, where we traverse the search space of neural network architectures with the help of an evolutionary algorithm which has been augmented with a novel approach of piecemeal-training. In contrast to the previously published NAS techniques, wherein the training with given data is considered an isolated task to estimate the performance of neural networks, our work demonstrates that a neural network architecture and the related weights can be jointly learned by combining concepts of the traditional training process and evolutionary architecture search in a single algorithm. The consolidation has been realised by breaking down the conventional training technique into smaller slices and collating them together with an integrated evolutionary architecture search algorithm. The constraints on architecture search space are placed by limiting its various parameters within a specified range of values, consequently regulating the neural network’s size and memory requirements. We validate this concept on two vastly different datasets, namely, the CIFAR-10 dataset in the domain of image classification, and PAMAP2 dataset in the Human Activity Recognition (HAR) domain. Starting from randomly initialized and untrained CNNs, the algorithm discovers models with competent architectures, which after complete training, reach an accuracy of of 92.5% for CIFAR-10 and 94.36% PAMAP2. We further extend the algorithm to include an additional conflicting search objective: the number of parameters of the neural network. Our multi-objective algorithm produces a Pareto optimal set of neural networks, by optimizing the search for both the accuracy and the parameter count, thus emphasizing the versatility of our approach.

scholarly journals Extensible Embedded Processor for Convolutional Neural Networks

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Joshua Misko ◽  
Shrikant S. Jadhav ◽  
Youngsoo Kim
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Search Algorithm ◽  
Recognition System ◽  
Synthetic Aperture ◽  
Embedded Processor ◽  
Computing Power ◽  
Execution Speed ◽  
Facial Recognition System

Convolutional neural networks (CNNs) require significant computing power during inference. Smart phones, for example, may not run a facial recognition system or search algorithm smoothly due to the lack of resources and supporting hardware. Methods for reducing memory size and increasing execution speed have been explored, but choosing effective techniques for an application requires extensive knowledge of the network architecture. This paper proposes a general approach to preparing a compressed deep neural network processor for inference with minimal additions to existing microprocessor hardware. To show the benefits to the proposed approach, an example CNN for synthetic aperture radar target classification is modified and complimentary custom processor instructions are designed. The modified CNN is examined to show the effects of the modifications and the custom processor instructions are profiled to illustrate the potential performance increase from the new extended instructions.

Sign in / Sign up

Export Citation Format

Share Document