scholarly journals ThriftyNets: Convolutional Neural Networks with Tiny Parameter Budget

IoT ◽  
2021 ◽  
Vol 2 (2) ◽  
pp. 222-235
Author(s):  
Guillaume Coiffier ◽  
Ghouthi Boukli Hacene ◽  
Vincent Gripon
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Spatial Resolution ◽  
Network Architecture ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Feature Maps ◽  
Neural Network Architecture

Deep Neural Networks are state-of-the-art in a large number of challenges in machine learning. However, to reach the best performance they require a huge pool of parameters. Indeed, typical deep convolutional architectures present an increasing number of feature maps as we go deeper in the network, whereas spatial resolution of inputs is decreased through downsampling operations. This means that most of the parameters lay in the final layers, while a large portion of the computations are performed by a small fraction of the total parameters in the first layers. In an effort to use every parameter of a network at its maximum, we propose a new convolutional neural network architecture, called ThriftyNet. In ThriftyNet, only one convolutional layer is defined and used recursively, leading to a maximal parameter factorization. In complement, normalization, non-linearities, downsamplings and shortcut ensure sufficient expressivity of the model. ThriftyNet achieves competitive performance on a tiny parameters budget, exceeding 91% accuracy on CIFAR-10 with less than 40 k parameters in total, 74.3% on CIFAR-100 with less than 600 k parameters, and 67.1% On ImageNet ILSVRC 2012 with no more than 4.15 M parameters. However, the proposed method typically requires more computations than existing counterparts.

scholarly journals Reynolds averaged turbulence modelling using deep neural networks with embedded invariance

2016 ◽  
Vol 807 ◽  
pp. 155-166 ◽  
Author(s):  
Julia Ling ◽  
Andrew Kurzawski ◽  
Jeremy Templeton
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Reynolds Stress ◽  
Network Architecture ◽  
Eddy Viscosity ◽  
Deep Neural Networks ◽  
Test Cases ◽  
Neural Network Architecture ◽  
Stress Anisotropy ◽  
Anisotropy Tensor

There exists significant demand for improved Reynolds-averaged Navier–Stokes (RANS) turbulence models that are informed by and can represent a richer set of turbulence physics. This paper presents a method of using deep neural networks to learn a model for the Reynolds stress anisotropy tensor from high-fidelity simulation data. A novel neural network architecture is proposed which uses a multiplicative layer with an invariant tensor basis to embed Galilean invariance into the predicted anisotropy tensor. It is demonstrated that this neural network architecture provides improved prediction accuracy compared with a generic neural network architecture that does not embed this invariance property. The Reynolds stress anisotropy predictions of this invariant neural network are propagated through to the velocity field for two test cases. For both test cases, significant improvement versus baseline RANS linear eddy viscosity and nonlinear eddy viscosity models is demonstrated.

EMOTIONS RECOGNITION IN HUMAN SPEECH USING DEEP NEURAL NETWORKS

2021 ◽  
pp. 44-51
Author(s):  
E. Yu. Shchetinin
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Recurrent Neural Network ◽  
Deep Neural Networks ◽  
Learning Algorithms ◽  
Machine Learning Algorithms ◽  
Audio Recordings ◽  
Computer Studies

The recognition of human emotions is one of the most relevant and dynamically developing areas of modern speech technologies, and the recognition of emotions in speech (RER) is the most demanded part of them. In this paper, we propose a computer model of emotion recognition based on an ensemble of bidirectional recurrent neural network with LSTM memory cell and deep convolutional neural network ResNet18. In this paper, computer studies of the RAVDESS database containing emotional speech of a person are carried out. RAVDESS-a data set containing 7356 files. Entries contain the following emotions: 0 – neutral, 1 – calm, 2 – happiness, 3 – sadness, 4 – anger, 5 – fear, 6 – disgust, 7 – surprise. In total, the database contains 16 classes (8 emotions divided into male and female) for a total of 1440 samples (speech only). To train machine learning algorithms and deep neural networks to recognize emotions, existing audio recordings must be pre-processed in such a way as to extract the main characteristic features of certain emotions. This was done using Mel-frequency cepstral coefficients, chroma coefficients, as well as the characteristics of the frequency spectrum of audio recordings. In this paper, computer studies of various models of neural networks for emotion recognition are carried out on the example of the data described above. In addition, machine learning algorithms were used for comparative analysis. Thus, the following models were trained during the experiments: logistic regression (LR), classifier based on the support vector machine (SVM), decision tree (DT), random forest (RF), gradient boosting over trees – XGBoost, convolutional neural network CNN, recurrent neural network RNN (ResNet18), as well as an ensemble of convolutional and recurrent networks Stacked CNN-RNN. The results show that neural networks showed much higher accuracy in recognizing and classifying emotions than the machine learning algorithms used. Of the three neural network models presented, the CNN + BLSTM ensemble showed higher accuracy.

scholarly journals A Novel Multi Hidden Layer Convolutional Neural Network for Content Based Image Retrieval

2020 ◽  
Vol 9 (3) ◽  
pp. 365-370
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Image Retrieval ◽  
Convolutional Neural Network ◽  
Network Architecture ◽  
Learning Algorithm ◽  
Machine Learning Algorithms ◽  
Content Based Image Retrieval ◽  
Neural Network Architecture ◽  
Query Image

The applications of a content-based image retrieval system in fields such as multimedia, security, medicine, and entertainment, have been implemented on a huge real-time database by using a convolutional neural network architecture. In general, thus far, content-based image retrieval systems have been implemented with machine learning algorithms. A machine learning algorithm is applicable to a limited database because of the few feature extraction hidden layers between the input and the output layers. The proposed convolutional neural network architecture was successfully implemented using 128 convolutional layers, pooling layers, rectifier linear unit (ReLu), and fully connected layers. A convolutional neural network architecture yields better results of its ability to extract features from an image. The Euclidean distance metric is used for calculating the similarity between the query image and the database images. It is implemented using the COREL database. The proposed system is successfully evaluated using precision, recall, and F-score. The performance of the proposed method is evaluated using the precision and recall.

scholarly journals Deep Learning-Based Feature Silencing for Accurate Concrete Crack Detection

Sensors ◽  
2020 ◽  
Vol 20 (16) ◽  
pp. 4403
Author(s):  
Umme Hafsa Billah ◽  
Hung Manh La ◽  
Alireza Tavakkoli
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Network Architecture ◽  
Crack Detection ◽  
Inspection System ◽  
Detection Accuracy ◽  
Feature Maps ◽  
Neural Network Architecture ◽  
Concrete Crack ◽  
Discriminative Feature

An autonomous concrete crack inspection system is necessary for preventing hazardous incidents arising from deteriorated concrete surfaces. In this paper, we present a concrete crack detection framework to aid the process of automated inspection. The proposed approach employs a deep convolutional neural network architecture for crack segmentation, while addressing the effect of gradient vanishing problem. A feature silencing module is incorporated in the proposed framework, capable of eliminating non-discriminative feature maps from the network to improve performance. Experimental results support the benefit of incorporating feature silencing within a convolutional neural network architecture for improving the network’s robustness, sensitivity, and specificity. An added benefit of the proposed architecture is its ability to accommodate for the trade-off between specificity (positive class detection accuracy) and sensitivity (negative class detection accuracy) with respect to the target application. Furthermore, the proposed framework achieves a high precision rate and processing time than the state-of-the-art crack detection architectures.

Application of convolutional neural networks for monitoring of marine objects

2020 ◽  
pp. 53-61
Author(s):  
Н.А. Полковникова ◽  
Е.В. Тузинкевич ◽  
А.Н. Попов
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computer Vision ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Research Process ◽  
Probability Of Detection ◽  
Neural Network Architecture ◽  
Deep Convolutional Neural Networks

В статье рассмотрены технологии компьютерного зрения на основе глубоких свёрточных нейронных сетей. Применение нейронных сетей особенно эффективно для решения трудно формализуемых задач. Разработана архитектура свёрточной нейронной сети применительно к задаче распознавания и классификации морских объектов на изображениях. В ходе исследования выполнен ретроспективный анализ технологий компьютерного зрения и выявлен ряд проблем, связанных с применением нейронных сетей: «исчезающий» градиент, переобучение и вычислительная сложность. При разработке архитектуры нейросети предложено использовать функцию активации RELU, обучение некоторых случайно выбранных нейронов и нормализацию с целью упрощения архитектуры нейросети. Сравнение используемых в нейросети функций активации ReLU, LeakyReLU, Exponential ReLU и SOFTMAX выполнено в среде Matlab R2020a. На основе свёрточной нейронной сети разработана программа на языке программирования Visual C# в среде MS Visual Studio для распознавания морских объектов. Программапредназначена для автоматизированной идентификации морских объектов, производит детектирование (нахождение объектов на изображении) и распознавание объектов с высокой вероятностью обнаружения. The article considers computer vision technologies based on deep convolutional neural networks. Application of neural networks is particularly effective for solving difficult formalized problems. As a result convolutional neural network architecture to the problem of recognition and classification of marine objects on images is implemented. In the research process a retrospective analysis of computer vision technologies was performed and a number of problems associated with the use of neural networks were identified: vanishing gradient, overfitting and computational complexity. To solve these problems in neural network architecture development, it was proposed to use RELU activation function, training some randomly selected neurons and normalization for simplification of neural network architecture. Comparison of ReLU, LeakyReLU, Exponential ReLU, and SOFTMAX activation functions used in the neural network implemented in Matlab R2020a.The computer program based on convolutional neural network for marine objects recognition implemented in Visual C# programming language in MS Visual Studio integrated development environment. The program is designed for automated identification of marine objects, produces detection (i.e., presence of objects on image), and objects recognition with high probability of detection.

scholarly journals AMBIENT: Accelerated Convolutional Neural Network Architecture Search for Regulatory Genomics

2021 ◽  
Author(s):  
Zijun Zhang ◽  
Evan M. Cofer ◽  
Olga G. Troyanskaya
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Network Architecture ◽  
Environmental Issue ◽  
Biological Sequences ◽  
Neural Network Architecture ◽  
Computing Power ◽  
Neural Architecture

Convolutional neural networks (CNN) have become a standard approach for modeling genomic sequences. CNNs can be effectively built by Neural Architecture Search (NAS) by trading computing power for accurate neural architectures. Yet, the consumption of immense computing power is a major practical, financial, and environmental issue for deep learning. Here, we present a novel NAS framework, AMBIENT, that generates highly accurate CNN architectures for biological sequences of diverse functions, while substantially reducing the computing cost of conventional NAS.

scholarly journals Using a Large Margin Context-Aware Convolutional Neural Network to Automatically Extract Disease-Disease Association from Literature: Comparative Analytic Study

10.2196/14502 ◽  
2019 ◽  
Vol 7 (4) ◽  
pp. e14502
Author(s):  
Po-Ting Lai ◽  
Wei-Liang Lu ◽  
Ting-Rung Kuo ◽  
Chia-Ru Chung ◽  
Jen-Chieh Han ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Support Vector Machine ◽  
Convolutional Neural Network ◽  
Network Architecture ◽  
Disease Association ◽  
Support Vector ◽  
Context Aware ◽  
Neural Network Architecture ◽  
Large Margin

Background Research on disease-disease association (DDA), like comorbidity and complication, provides important insights into disease treatment and drug discovery, and a large body of the literature has been published in the field. However, using current search tools, it is not easy for researchers to retrieve information on the latest DDA findings. First, comorbidity and complication keywords pull up large numbers of PubMed studies. Second, disease is not highlighted in search results. Finally, DDA is not identified, as currently no disease-disease association extraction (DDAE) dataset or tools are available. Objective As there are no available DDAE datasets or tools, this study aimed to develop (1) a DDAE dataset and (2) a neural network model for extracting DDA from the literature. Methods In this study, we formulated DDAE as a supervised machine learning classification problem. To develop the system, we first built a DDAE dataset. We then employed two machine learning models, support vector machine and convolutional neural network, to extract DDA. Furthermore, we evaluated the effect of using the output layer as features of the support vector machine-based model. Finally, we implemented large margin context-aware convolutional neural network architecture to integrate context features and convolutional neural networks through the large margin function. Results Our DDAE dataset consisted of 521 PubMed abstracts. Experiment results showed that the support vector machine-based approach achieved an F1 measure of 80.32%, which is higher than the convolutional neural network-based approach (73.32%). Using the output layer of convolutional neural network as a feature for the support vector machine does not further improve the performance of support vector machine. However, our large margin context-aware-convolutional neural network achieved the highest F1 measure of 84.18% and demonstrated that combining the hinge loss function of support vector machine with a convolutional neural network into a single neural network architecture outperforms other approaches. Conclusions To facilitate the development of text-mining research for DDAE, we developed the first publicly available DDAE dataset consisting of disease mentions, Medical Subject Heading IDs, and relation annotations. We developed different conventional machine learning models and neural network architectures and evaluated their effects on our DDAE dataset. To further improve DDAE performance, we propose an large margin context-aware-convolutional neural network model for DDAE that outperforms other approaches.

scholarly journals Fast Pre-Diagnosis of Neoplastic Changes in Cytology Images Using Machine Learning

2021 ◽  
Vol 11 (16) ◽  
pp. 7181
Author(s):  
Jakub Caputa ◽  
Daria Łukasik ◽  
Maciej Wielgosz ◽  
Michał Karwatowski ◽  
Rafał Frączek ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Network Architecture ◽  
State Of The Art ◽  
Prototype Model ◽  
Model Parameters ◽  
Round Cell ◽  
Neural Network Architecture ◽  
Original Dataset ◽  
Automated Tools

We present the experiment results to use the YOLOv3 neural network architecture to automatically detect tumor cells in cytological samples taken from the skin in canines. A rich dataset of 1219 smeared sample images with 28,149 objects was gathered and annotated by the vet doctor to perform the experiments. It covers three types of common round cell neoplasms: mastocytoma, histiocytoma, and lymphoma. The dataset has been thoroughly described in the paper and is publicly available. The YOLOv3 neural network architecture was trained using various schemes involving original dataset modification and the different model parameters. The experiments showed that the prototype model achieved 0.7416 mAP, which outperforms the state-of-the-art machine learning and human estimated results. We also provided a series of analyses that may facilitate ML-based solutions by casting more light on some aspects of its performance. We also presented the main discrepancies between ML-based and human-based diagnoses. This outline may help depict the scenarios and how the automated tools may support the diagnosis process.

scholarly journals Deep neo-fuzzy neural network and its learning

2019 ◽  
Vol 1 (92) ◽  
pp. 3-8
Author(s):  
E.V. Bodyansky ◽  
Т.Е. Antonenko
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Fuzzy Neural Network ◽  
Deep Neural Networks ◽  
Neural Network Architecture ◽  
The Neural Network ◽  
Fuzzy Neural ◽  
Main Component ◽  
Triangular Membership Function

Optimizing the learning speedof deep neural networks is an extremely important issue. Modern approaches focus on the use of neural networksbased on the Rosenblatt perceptron. But the results obtained are not satisfactory for industrial and scientific needs inthe context of the speed of learning neural networks. Also, this approach stumbles upon the problems of a vanishingand exploding gradient. To solve the problem, the paper proposed using a neo-fuzzy neuron, whose properties arebased on the F-transform. The article discusses the use of neo-fuzzy neuron as the main component of the neuralnetwork. The architecture of a deep neo-fuzzy neural network is shown, as well as a backpropagation algorithmfor this architecture with a triangular membership function for neo-fuzzy neuron. The main advantages of usingneo-fuzzy neuron as the main component of the neural network are given. The article describes the properties of aneo-fuzzy neuron that addresses the issues of improving speed and vanishing or exploding gradient. The proposedneo-fuzzy deep neural network architecture is compared with standard deep networks based on the Rosenblattperceptron.

Sign in / Sign up

Export Citation Format

Share Document