scholarly journals Implications of Pooling Strategies in Convolutional Neural Networks: A Deep Insight

2019 ◽  
Vol 44 (3) ◽  
pp. 303-330 ◽  
Author(s):  
Shallu Sharma ◽  
Rajesh Mehra
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Computational Cost ◽  
Activation Function ◽  
Training Time ◽  
Pooling Strategies ◽  
Deep Cnn ◽  
And Training

Abstract Convolutional neural networks (CNN) is a contemporary technique for computer vision applications, where pooling implies as an integral part of the deep CNN. Besides, pooling provides the ability to learn invariant features and also acts as a regularizer to further reduce the problem of overfitting. Additionally, the pooling techniques significantly reduce the computational cost and training time of networks which are equally important to consider. Here, the performances of pooling strategies on different datasets are analyzed and discussed qualitatively. This study presents a detailed review of the conventional and the latest strategies which would help in appraising the readers with the upsides and downsides of each strategy. Also, we have identified four fundamental factors namely network architecture, activation function, overlapping and regularization approaches which immensely affect the performance of pooling operations. It is believed that this work would help in extending the scope of understanding the significance of CNN along with pooling regimes for solving computer vision problems.

scholarly journals AN APPROACH TO ACCELERATE THE TRAINING OF CONVOLUTIONAL NEURAL NETWORKS BY TUNING THE HYPERPARAMETERS OF LEARNING

2020 ◽  
Vol 2 (2) ◽  
pp. 32-37
Author(s):  
P. RADIUK ◽  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Video Processing ◽  
Network Architecture ◽  
Random Search ◽  
Machine Learning Algorithms ◽  
Language Recognition ◽  
Data Set ◽  
Training Time

Over the last decade, a set of machine learning algorithms called deep learning has led to significant improvements in computer vision, natural language recognition and processing. This has led to the widespread use of a variety of commercial, learning-based products in various fields of human activity. Despite this success, the use of deep neural networks remains a black box. Today, the process of setting hyperparameters and designing a network architecture requires experience and a lot of trial and error and is based more on chance than on a scientific approach. At the same time, the task of simplifying deep learning is extremely urgent. To date, no simple ways have been invented to establish the optimal values of learning hyperparameters, namely learning speed, sample size, data set, learning pulse, and weight loss. Grid search and random search of hyperparameter space are extremely resource intensive. The choice of hyperparameters is critical for the training time and the final result. In addition, experts often choose one of the standard architectures (for example, ResNets and ready-made sets of hyperparameters. However, such kits are usually suboptimal for specific practical tasks. The presented work offers an approach to finding the optimal set of hyperparameters of learning ZNM. An integrated approach to all hyperparameters is valuable because there is an interdependence between them. The aim of the work is to develop an approach for setting a set of hyperparameters, which will reduce the time spent during the design of ZNM and ensure the efficiency of its work. In recent decades, the introduction of deep learning methods, in particular convolutional neural networks (CNNs), has led to impressive success in image and video processing. However, the training of CNN has been commonly mostly based on the employment of quasi-optimal hyperparameters. Such an approach usually requires huge computational and time costs to train the network and does not guarantee a satisfactory result. However, hyperparameters play a crucial role in the effectiveness of CNN, as diverse hyperparameters lead to models with significantly different characteristics. Poorly selected hyperparameters generally lead to low model performance. The issue of choosing optimal hyperparameters for CNN has not been resolved yet. The presented work proposes several practical approaches to setting hyperparameters, which allows reducing training time and increasing the accuracy of the model. The article considers the function of training validation loss during underfitting and overfitting. There are guidelines in the end to reach the optimization point. The paper also considers the regulation of learning rate and momentum to accelerate network training. All experiments are based on the widespread CIFAR-10 and CIFAR-100 datasets.

Application of convolutional neural networks for monitoring of marine objects

2020 ◽  
pp. 53-61
Author(s):  
Н.А. Полковникова ◽  
Е.В. Тузинкевич ◽  
А.Н. Попов
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computer Vision ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Research Process ◽  
Probability Of Detection ◽  
Neural Network Architecture ◽  
Deep Convolutional Neural Networks

В статье рассмотрены технологии компьютерного зрения на основе глубоких свёрточных нейронных сетей. Применение нейронных сетей особенно эффективно для решения трудно формализуемых задач. Разработана архитектура свёрточной нейронной сети применительно к задаче распознавания и классификации морских объектов на изображениях. В ходе исследования выполнен ретроспективный анализ технологий компьютерного зрения и выявлен ряд проблем, связанных с применением нейронных сетей: «исчезающий» градиент, переобучение и вычислительная сложность. При разработке архитектуры нейросети предложено использовать функцию активации RELU, обучение некоторых случайно выбранных нейронов и нормализацию с целью упрощения архитектуры нейросети. Сравнение используемых в нейросети функций активации ReLU, LeakyReLU, Exponential ReLU и SOFTMAX выполнено в среде Matlab R2020a. На основе свёрточной нейронной сети разработана программа на языке программирования Visual C# в среде MS Visual Studio для распознавания морских объектов. Программапредназначена для автоматизированной идентификации морских объектов, производит детектирование (нахождение объектов на изображении) и распознавание объектов с высокой вероятностью обнаружения. The article considers computer vision technologies based on deep convolutional neural networks. Application of neural networks is particularly effective for solving difficult formalized problems. As a result convolutional neural network architecture to the problem of recognition and classification of marine objects on images is implemented. In the research process a retrospective analysis of computer vision technologies was performed and a number of problems associated with the use of neural networks were identified: vanishing gradient, overfitting and computational complexity. To solve these problems in neural network architecture development, it was proposed to use RELU activation function, training some randomly selected neurons and normalization for simplification of neural network architecture. Comparison of ReLU, LeakyReLU, Exponential ReLU, and SOFTMAX activation functions used in the neural network implemented in Matlab R2020a.The computer program based on convolutional neural network for marine objects recognition implemented in Visual C# programming language in MS Visual Studio integrated development environment. The program is designed for automated identification of marine objects, produces detection (i.e., presence of objects on image), and objects recognition with high probability of detection.

scholarly journals Recognition of Handwritten Digit using Convolutional Neural Network (CNN)

2019 ◽  
pp. 27-33 ◽  
Author(s):  
Md. Anwar Hossain ◽  
Md. Mohon Ali
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computer Vision ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Human Vision ◽  
Training Time ◽  
The World ◽  
Handwritten Digit

Humans can see and visually sense the world around them by using their eyes and brains. Computer vision works on enabling computers to see and process images in the same way that human vision does. Several algorithms developed in the area of computer vision to recognize images. The goal of our work will be to create a model that will be able to identify and determine the handwritten digit from its image with better accuracy. We aim to complete this by using the concepts of Convolutional Neural Network and MNIST dataset. We will also show how MatConvNet can be used to implement our model with CPU training as well as less training time. Though the goal is to create a model which can recognize the digits, we can extend it for letters and then a person’s handwriting. Through this work, we aim to learn and practically apply the concepts of Convolutional Neural Networks.

scholarly journals Multi-Path Deep CNN with Residual Inception Network for Single Image Super-Resolution

Electronics ◽  
2021 ◽  
Vol 10 (16) ◽  
pp. 1979
Author(s):  
Wazir Muhammad ◽  
Zuhaibuddin Bhutto ◽  
Arslan Ansari ◽  
Mudasar Latif Memon ◽  
Ramesh Kumar ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Computational Cost ◽  
Similarity Index ◽  
Super Resolution ◽  
Single Image ◽  
Index Matrix ◽  
Deep Cnn ◽  
Image Super Resolution ◽  
Single Image Super Resolution

Recent research on single-image super-resolution (SISR) using deep convolutional neural networks has made a breakthrough and achieved tremendous performance. Despite their significant progress, numerous convolutional neural networks (CNN) are limited in practical applications, owing to the requirement of the heavy computational cost of the model. This paper proposes a multi-path network for SISR, known as multi-path deep CNN with residual inception network for single image super-resolution. In detail, a residual/ResNet block with an Inception block supports the main framework of the entire network architecture. In addition, remove the batch normalization layer from the residual network (ResNet) block and max-pooling layer from the Inception block to further reduce the number of parameters to preventing the over-fitting problem during the training. Moreover, a conventional rectified linear unit (ReLU) is replaced with Leaky ReLU activation function to speed up the training process. Specifically, we propose a novel two upscale module, which adopts three paths to upscale the features by jointly using deconvolution and upsampling layers, instead of using single deconvolution layer or upsampling layer alone. The extensive experimental results on image super-resolution (SR) using five publicly available test datasets, which show that the proposed model not only attains the higher score of peak signal-to-noise ratio/structural similarity index matrix (PSNR/SSIM) but also enables faster and more efficient calculations against the existing image SR methods. For instance, we improved our method in terms of overall PSNR on the SET5 dataset with challenging upscale factor 8× as 1.88 dB over the baseline bicubic method and reduced computational cost in terms of number of parameters 62% by deeply-recursive convolutional neural network (DRCN) method.

scholarly journals Face Recognition Based on Lightweight Convolutional Neural Networks

Information ◽  
2021 ◽  
Vol 12 (5) ◽  
pp. 191
Author(s):  
Wenting Liu ◽  
Li Zhou ◽  
Jie Chen
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Face Recognition ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Recognition Task ◽  
Training Method ◽  
The Face ◽  
And Storage ◽  
And Training

Face recognition algorithms based on deep learning methods have become increasingly popular. Most of these are based on highly precise but complex convolutional neural networks (CNNs), which require significant computing resources and storage, and are difficult to deploy on mobile devices or embedded terminals. In this paper, we propose several methods to improve the algorithms for face recognition based on a lightweight CNN, which is further optimized in terms of the network architecture and training pattern on the basis of MobileFaceNet. Regarding the network architecture, we introduce the Squeeze-and-Excitation (SE) block and propose three improved structures via a channel attention mechanism—the depthwise SE module, the depthwise separable SE module, and the linear SE module—which are able to learn the correlation of information between channels and assign them different weights. In addition, a novel training method for the face recognition task combined with an additive angular margin loss function is proposed that performs the compression and knowledge transfer of the deep network for face recognition. Finally, we obtained high-precision and lightweight face recognition models with fewer parameters and calculations that are more suitable for applications. Through extensive experiments and analysis, we demonstrate the effectiveness of the proposed methods.

IMPROVING FACE RECOGNITION MODELS USING CONVOLUTIONAL NEURAL NETWORKS, METRIC LEARNING AND OPTIMIZATION METHOD

2021 ◽  
Vol 5 ◽  
pp. 140-158
Author(s):  
Andrey Litvynchuk ◽  
◽  
Lesya Baranovska ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computer Vision ◽  
Face Recognition ◽  
Comparative Analysis ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Metric Learning ◽  
Stochastic Gradient Descent ◽  
Google Search

Face recognition is one of the main tasks of computer vision. It has many applications, which has led to a huge amount of research in this area. And although research in the field has been going on since the beginning of the computer vision, good results could be achieved only with the help of convolutional neural networks. In this work, a comparative analysis of facial recognition methods before convolutional neural networks was performed. A set of neural network architectures, methods of metric learning and optimization are considered. There were performed bunch of experiments and comparative analysis of the considered methods of improvement of convolutional neural networks. As a result a universal algorithm for training the face recognition model was obtained. To compare different approaches of face recognition, we chose a dataset called VGGFace2. It consists of 3,31 million images of 9131 people. It was created using images from the Google search engine. Initially, pre-trained neural networks were used to select photographs with humans. The images were then checked mannualy. For the validation sample, we set aside 50 images of 500 people, for a total of 25,000 images. Almost all experiments were performed iteratively. For example, we choose the best optimizer and then we use it to search for best arctitecture. As expected, neural networks with more parameters and more sophisticated architecture showed better results in this task. Among the considered models the best was Se-ResNet50. Metric learning is a method by which it is possible to achieve good accuracy in face recognition. Without this method it would be impossible to solve the problem. To optimize neural networks, we considered both adaptive and simple optimizers. It turned out that the stochastic gradient descent with moment is the best for this problem, and adaptive methods showed a rather poor result. In general, using different approaches, we were able to obtain an accuracy of 92 %, which is 25,5 % better than the baseline experiment. We see next ways for the further development of the research subject: improving neural network architecture, collecting more data and applying better regularization techniques.

scholarly journals PLANET: Improved Convolutional Neural Networks with Image Enhancement for Image Classification

2020 ◽  
Vol 2020 ◽  
pp. 1-10 ◽  
Author(s):  
Chaohui Tang ◽  
Qingxin Zhu ◽  
Wenjun Wu ◽  
Wenlin Huang ◽  
Chaoqun Hong ◽  
...  
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Image Classification ◽  
Image Enhancement ◽  
Convolutional Neural Networks ◽  
Image Data ◽  
Training Samples ◽  
Enhancement Method ◽  
Deep Cnn ◽  
Classification Tasks

In the past few years, deep learning has become a research hotspot and has had a profound impact on computer vision. Deep CNN has been proven to be the most important and effective model for image processing, but due to the lack of training samples and huge number of learning parameters, it is easy to tend to overfit. In this work, we propose a new two-stage CNN image classification network, named “Improved Convolutional Neural Networks with Image Enhancement for Image Classification” and PLANET in abbreviation, which uses a new image data enhancement method called InnerMove to enhance images and augment the number of training samples. InnerMove is inspired by the “object movement” scene in computer vision and can improve the generalization ability of deep CNN models for image classification tasks. Sufficient experiment results show that PLANET utilizing InnerMove for image enhancement outperforms the comparative algorithms, and InnerMove has a more significant effect than the comparative data enhancement methods for image classification tasks.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

scholarly journals Evaluation of Mixed Deep Neural Networks for Reverberant Speech Enhancement

Biomimetics ◽  
2019 ◽  
Vol 5 (1) ◽  
pp. 1 ◽  
Author(s):  
Michelle Gutiérrez-Muñoz ◽  
Astryd González-Salazar ◽  
Marvin Coto-Jiménez
Keyword(s):  
Neural Networks ◽  
Short Term Memory ◽  
Computational Cost ◽  
Real Life ◽  
Fixed Number ◽  
Training Procedure ◽  
Statistical Validation ◽  
Significant Drop ◽  
Training Time ◽  
Important Solution

Speech signals are degraded in real-life environments, as a product of background noise or other factors. The processing of such signals for voice recognition and voice analysis systems presents important challenges. One of the conditions that make adverse quality difficult to handle in those systems is reverberation, produced by sound wave reflections that travel from the source to the microphone in multiple directions. To enhance signals in such adverse conditions, several deep learning-based methods have been proposed and proven to be effective. Recently, recurrent neural networks, especially those with long short-term memory (LSTM), have presented surprising results in tasks related to time-dependent processing of signals, such as speech. One of the most challenging aspects of LSTM networks is the high computational cost of the training procedure, which has limited extended experimentation in several cases. In this work, we present a proposal to evaluate the hybrid models of neural networks to learn different reverberation conditions without any previous information. The results show that some combinations of LSTM and perceptron layers produce good results in comparison to those from pure LSTM networks, given a fixed number of layers. The evaluation was made based on quality measurements of the signal’s spectrum, the training time of the networks, and statistical validation of results. In total, 120 artificial neural networks of eight different types were trained and compared. The results help to affirm the fact that hybrid networks represent an important solution for speech signal enhancement, given that reduction in training time is on the order of 30%, in processes that can normally take several days or weeks, depending on the amount of data. The results also present advantages in efficiency, but without a significant drop in quality.

Sign in / Sign up

Export Citation Format

Share Document