Deep Architecture of Neural Networks Using Stochastic Integral Computation in VLSI

Abstract Deep Neural Networks (DNNs) have shown great success in many fields. Various network architectures have been developed for different applications. Regardless of the complexities of the networks, DNNs do not provide model uncertainty. Bayesian Neural Networks (BNNs), on the other hand, is able to make probabilistic inference. Among various types of BNNs, Dropout as a Bayesian Approximation converts a Neural Network (NN) to a BNN by adding a dropout layer after each weight layer in the NN. This technique provides a simple transformation from a NN to a BNN. However, for DNNs, adding a dropout layer to each weight layer would lead to a strong regularization due to the deep architecture. Previous researches [1, 2, 3] have shown that adding a dropout layer after each weight layer in a DNN is unnecessary. However, how to place dropout layers in a ResNet for regression tasks are less explored. In this work, we perform an empirical study on how different dropout placements would affect the performance of a Bayesian DNN. We use a regression model modified from ResNet as the DNN and place the dropout layers at different places in the regression ResNet. Our experimental results show that it is not necessary to add a dropout layer after every weight layer in the Regression ResNet to let it be able to make Bayesian Inference. Placing Dropout layers between the stacked blocks i.e. Dense+Identity+Identity blocks has the best performance in Predictive Interval Coverage Probability (PICP). Placing a dropout layer after each stacked block has the best performance in Root Mean Square Error (RMSE).

Download Full-text

A CONVblock for Convolutional Neural Networks

Deep Learning Applications in Medical Imaging - Advances in Medical Technologies and Clinical Practice ◽

10.4018/978-1-7998-5071-7.ch004 ◽

2021 ◽

pp. 100-113

Author(s):

Hmidi Alaeddine ◽

Malek Jihene

Keyword(s):

Neural Networks ◽

Image Classification ◽

Convolutional Neural Networks ◽

Experimental Results ◽

Classification Models ◽

Deep Architecture ◽

Improved Performance ◽

Convolution Filters ◽

Classification Database

The reduction in the size of convolution filters has been shown to be effective in image classification models. They make it possible to reduce the calculation and the number of parameters used in the operations of the convolution layer while increasing the efficiency of the representation. The authors present a deep architecture for classification with improved performance. The main objective of this architecture is to improve the main performances of the network thanks to a new design based on CONVblock. The proposal is evaluated on a classification database: CIFAR-10 and MNIST. The experimental results demonstrate the effectiveness of the proposed method. This architecture offers an error of 1.4% on CIFAR-10 and 0.055% on MNIST.

Download Full-text

Deep Learning for Feature Extraction in Remote Sensing: A Case-Study of Aerial Scene Classification

Sensors ◽

10.3390/s20143906 ◽

2020 ◽

Vol 20 (14) ◽

pp. 3906 ◽

Cited By ~ 6

Author(s):

Biserka Petrovska ◽

Eftim Zdravevski ◽

Petre Lameski ◽

Roberto Corizzo ◽

Ivan Štajduhar ◽

...

Keyword(s):

Remote Sensing ◽

Neural Networks ◽

Feature Extraction ◽

Deep Learning ◽

Aerial Images ◽

Support Vector ◽

Scene Classification ◽

Network Layers ◽

Deep Architecture ◽

Real World Datasets

Scene classification relying on images is essential in many systems and applications related to remote sensing. The scientific interest in scene classification from remotely collected images is increasing, and many datasets and algorithms are being developed. The introduction of convolutional neural networks (CNN) and other deep learning techniques contributed to vast improvements in the accuracy of image scene classification in such systems. To classify the scene from areal images, we used a two-stream deep architecture. We performed the first part of the classification, the feature extraction, using pre-trained CNN that extracts deep features of aerial images from different network layers: the average pooling layer or some of the previous convolutional layers. Next, we applied feature concatenation on extracted features from various neural networks, after dimensionality reduction was performed on enormous feature vectors. We experimented extensively with different CNN architectures, to get optimal results. Finally, we used the Support Vector Machine (SVM) for the classification of the concatenated features. The competitiveness of the examined technique was evaluated on two real-world datasets: UC Merced and WHU-RS. The obtained classification accuracies demonstrate that the considered method has competitive results compared to other cutting-edge techniques.

Download Full-text

A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks

Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization - UMAP '17 ◽

10.1145/3079628.3079684 ◽

2017 ◽

Cited By ~ 20

Author(s):

Alessandro Suglia ◽

Claudio Greco ◽

Cataldo Musto ◽

Marco de Gemmis ◽

Pasquale Lops ◽

...

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Deep Architecture

Download Full-text

Multiobjective and categorical global optimization of photonic structures based on ResNet generative neural networks

Nanophotonics ◽

10.1515/nanoph-2020-0407 ◽

2020 ◽

Vol 10 (1) ◽

pp. 361-369 ◽

Cited By ~ 1

Author(s):

Jiaqi Jiang ◽

Jonathan A. Fan

Keyword(s):

Neural Networks ◽

Global Optimization ◽

Photonic Devices ◽

Global Optimum ◽

Proof Of Concept ◽

Deep Architecture ◽

Complex Design ◽

Light Filters ◽

Multiple Material ◽

Full Design

AbstractWe show that deep generative neural networks, based on global optimization networks (GLOnets), can be configured to perform the multiobjective and categorical global optimization of photonic devices. A residual network scheme enables GLOnets to evolve from a deep architecture, which is required to properly search the full design space early in the optimization process, to a shallow network that generates a narrow distribution of globally optimal devices. As a proof-of-concept demonstration, we adapt our method to design thin-film stacks consisting of multiple material types. Benchmarks with known globally optimized antireflection structures indicate that GLOnets can find the global optimum with orders of magnitude faster speeds compared to conventional algorithms. We also demonstrate the utility of our method in complex design tasks with its application to incandescent light filters. These results indicate that advanced concepts in deep learning can push the capabilities of inverse design algorithms for photonics.

Download Full-text

Deep Self-Organizing Map of Convolutional Layers for Clustering and Visualizing Image Data

Machine Learning and Knowledge Extraction ◽

10.3390/make3040044 ◽

2021 ◽

Vol 3 (4) ◽

pp. 879-899

Author(s):

Christos Ferles ◽

Yannis Papanikolaou ◽

Stylianos P. Savaidis ◽

Stelios A. Mitilineos

Keyword(s):

Neural Networks ◽

Image Data ◽

Building Blocks ◽

The Self ◽

Self Organizing Map ◽

Self Organizing Maps ◽

Deep Architecture ◽

Unsupervised Deep Learning ◽

Deep Learning Model ◽

Self Organizing

The self-organizing convolutional map (SOCOM) hybridizes convolutional neural networks, self-organizing maps, and gradient backpropagation optimization into a novel integrated unsupervised deep learning model. SOCOM structurally combines, architecturally stacks, and algorithmically fuses its deep/unsupervised learning components. The higher-level representations produced by its underlying convolutional deep architecture are embedded in its topologically ordered neural map output. The ensuing unsupervised clustering and visualization operations reflect the model’s degree of synergy between its building blocks and synopsize its range of applications. Clustering results are reported on the STL-10 benchmark dataset coupled with the devised neural map visualizations. The series of conducted experiments utilize a deep VGG-based SOCOM model.

Download Full-text

Capturing Discriminative Information Using a Deep Architecture in Acoustic Scene Classification

Applied Sciences ◽

10.3390/app11188361 ◽

2021 ◽

Vol 11 (18) ◽

pp. 8361

Author(s):

Hye-jin Shim ◽

Jee-weon Jung ◽

Ju-ho Kim ◽

Ha-jin Yu

Keyword(s):

Neural Networks ◽

Deep Neural Networks ◽

Data Augmentation ◽

Acoustic Properties ◽

Discriminative Power ◽

Activation Functions ◽

Scene Classification ◽

Feature Map ◽

Deep Architecture

Acoustic scene classification contains frequently misclassified pairs of classes that share many common acoustic properties. Specific details can provide vital clues for distinguishing such pairs of classes. However, these details are generally not noticeable and are hard to generalize for different data distributions. In this study, we investigate various methods for capturing discriminative information and simultaneously improve the generalization ability. We adopt a max feature map method that replaces conventional non-linear activation functions in deep neural networks; therefore, we apply an element-wise comparison between the different filters of a convolution layer’s output. Two data augmentation methods and two deep architecture modules are further explored to reduce overfitting and sustain the system’s discriminative power. Various experiments are conducted using the “detection and classification of acoustic scenes and events 2020 task1-a” dataset to validate the proposed methods. Our results show that the proposed system consistently outperforms the baseline, where the proposed system demonstrates an accuracy of 70.4% compared to the baseline at 65.1%.

Download Full-text