scholarly journals Extracting automata from neural networks using active learning

2021 ◽  
Vol 7 ◽  
pp. e436
Author(s):  
Zhiwu Xu ◽  
Cheng Wen ◽  
Shengchao Qin ◽  
Mengda He
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Active Learning ◽  
Block Size ◽  
Learning Framework ◽  
Passive Learning ◽  
The Neural Network ◽  
Equivalence Queries ◽  
Neural Network Classifiers

Deep learning is one of the most advanced forms of machine learning. Most modern deep learning models are based on an artificial neural network, and benchmarking studies reveal that neural networks have produced results comparable to and in some cases superior to human experts. However, the generated neural networks are typically regarded as incomprehensible black-box models, which not only limits their applications, but also hinders testing and verifying. In this paper, we present an active learning framework to extract automata from neural network classifiers, which can help users to understand the classifiers. In more detail, we use Angluin’s L* algorithm as a learner and the neural network under learning as an oracle, employing abstraction interpretation of the neural network for answering membership and equivalence queries. Our abstraction consists of value, symbol and word abstractions. The factors that may affect the abstraction are also discussed in the paper. We have implemented our approach in a prototype. To evaluate it, we have performed the prototype on a MNIST classifier and have identified that the abstraction with interval number 2 and block size 1 × 28 offers the best performance in terms of F1 score. We also have compared our extracted DFA against the DFAs learned via the passive learning algorithms provided in LearnLib and the experimental results show that our DFA gives a better performance on the MNIST dataset.

scholarly journals Increasing of Thermal Images Resolution Using Deep Learning Neural Networks

2021 ◽  
Vol 25 (3) ◽  
pp. 31-35
Author(s):  
Piotr Więcek ◽  
Dominik Sankowski
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Execution Time ◽  
High Accuracy ◽  
New Method ◽  
Residual Network ◽  
Thermal Images ◽  
The Neural Network

The article presents a new algorithm for increasing the resolution of thermal images. For this purpose, the residual network was integrated with the Kernel-Sharing Atrous Convolution (KSAC) image sub-sampling module. A significant reduction in the algorithm’s complexity and shortening the execution time while maintaining high accuracy were achieved. The neural network has been implemented in the PyTorch environment. The results of the proposed new method of increasing the resolution of thermal images with sizes 32 × 24, 160 × 120 and 640 × 480 for scales up to 6 are presented.

scholarly journals Convolutional neural network for image classification based on transfer learning technique

2021 ◽  
Author(s):  
Ghassan Mohammed Halawani
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Image Classification ◽  
Transfer Learning ◽  
Learning Networks ◽  
The Neural Network ◽  
Learning Technique ◽  
Common Architecture

The main purpose of this project is to modify a convolutional neural network for image classification, based on a deep-learning framework. A transfer learning technique is used by the MATLAB interface to Alex-Net to train and modify the parameters in the last two fully connected layers of Alex-Net with a new dataset to perform classifications of thousands of images. First, the general common architecture of most neural networks and their benefits are presented. The mathematical models and the role of each part in the neural network are explained in detail. Second, different neural networks are studied in terms of architecture, application, and the working method to highlight the strengths and weaknesses of each of neural network. The final part conducts a detailed study on one of the most powerful deep-learning networks in image classification – i.e. the convolutional neural network – and how it can be modified to suit different classification tasks by using transfer learning technique in MATLAB.

scholarly journals Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training

2021 ◽  
Vol 118 (43) ◽  
pp. e2103091118
Author(s):  
Cong Fang ◽  
Hangfeng He ◽  
Qi Long ◽  
Weijie J. Su
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Model Minority ◽  
Tight Frame ◽  
Learning Models ◽  
The Neural Network ◽  
Long Time ◽  
Topmost Layer

In this paper, we introduce the Layer-Peeled Model, a nonconvex, yet analytically tractable, optimization program, in a quest to better understand deep neural networks that are trained for a sufficiently long time. As the name suggests, this model is derived by isolating the topmost layer from the remainder of the neural network, followed by imposing certain constraints separately on the two parts of the network. We demonstrate that the Layer-Peeled Model, albeit simple, inherits many characteristics of well-trained neural networks, thereby offering an effective tool for explaining and predicting common empirical patterns of deep-learning training. First, when working on class-balanced datasets, we prove that any solution to this model forms a simplex equiangular tight frame, which, in part, explains the recently discovered phenomenon of neural collapse [V. Papyan, X. Y. Han, D. L. Donoho, Proc. Natl. Acad. Sci. U.S.A. 117, 24652–24663 (2020)]. More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto-unknown phenomenon that we term Minority Collapse, which fundamentally limits the performance of deep-learning models on the minority classes. In addition, we use the Layer-Peeled Model to gain insights into how to mitigate Minority Collapse. Interestingly, this phenomenon is first predicted by the Layer-Peeled Model before being confirmed by our computational experiments.

scholarly journals Solving Differential Equations Using Feedforward Neural Networks

2021 ◽  
pp. 385-399
Author(s):  
Wilson Guasti Junior ◽  
Isaac P. Santos
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Differential Equations ◽  
Feedforward Neural Networks ◽  
Activation Functions ◽  
Learning Models ◽  
Python Language ◽  
The Neural Network ◽  
Minimization Methods

Abstract In this work we explore the use of deep learning models based on deep feedforward neural networks to solve ordinary and partial differential equations. The illustration of this methodology is given by solving a variety of initial and boundary value problems. The numerical results, obtained based on different feedforward neural networks structures, activation functions and minimization methods, were compared to each other and to the exact solutions. The neural network was implemented using the Python language, with the Tensorflow library.

Development of Deep Learning Framework for Mathematical Morphology

2019 ◽  
Vol 33 (06) ◽  
pp. 1954024 ◽  
Author(s):  
Frank Y. Shih ◽  
Yucong Shen ◽  
Xin Zhong
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Mathematical Morphology ◽  
Experimental Results ◽  
Morphological Operations ◽  
Learning Framework ◽  
Structuring Element ◽  
The Neural Network ◽  
Object Features

Mathematical morphology has been applied as a collection of nonlinear operations related to object features in images. In this paper, we present morphological layers in deep learning framework, namely MorphNet, to perform atomic morphological operations, such as dilation and erosion. For propagation of losses through the proposed deep learning framework, we approximate the dilation and erosion operations by differential and smooth multivariable functions of the softmax function, and therefore enable the neural network to be optimized. The proposed operations are analyzed by the derivative of approximation functions in the deep learning framework. Experimental results show that the output structuring element of a morphological neuron and the target structuring element are matched to confirm the efficiency and correctness of the proposed framework.

scholarly journals Automatic Defects Segmentation and Identification by Deep Learning Algorithm with Pulsed Thermography: Synthetic and Experimental Data

2021 ◽  
Vol 5 (1) ◽  
pp. 9
Author(s):  
Qiang Fang ◽  
Clemente Ibarra-Castanedo ◽  
Xavier Maldague
Keyword(s):  
Neural Network ◽  
Experimental Data ◽  
Neural Networks ◽  
Deep Learning ◽  
Infrared Thermography ◽  
Learning Algorithm ◽  
Synthetic Data ◽  
Training Data ◽  
Experimental Database ◽  
The Neural Network

In quality evaluation (QE) of the industrial production field, infrared thermography (IRT) is one of the most crucial techniques used for evaluating composite materials due to the properties of low cost, fast inspection of large surfaces, and safety. The application of deep neural networks tends to be a prominent direction in IRT Non-Destructive Testing (NDT). During the training of the neural network, the Achilles heel is the necessity of a large database. The collection of huge amounts of training data is the high expense task. In NDT with deep learning, synthetic data contributing to training in infrared thermography remains relatively unexplored. In this paper, synthetic data from the standard Finite Element Models are combined with experimental data to build repositories with Mask Region based Convolutional Neural Networks (Mask-RCNN) to strengthen the neural network, learning the essential features of objects of interest and achieving defect segmentation automatically. These results indicate the possibility of adapting inexpensive synthetic data merging with a certain amount of the experimental database for training the neural networks in order to achieve the compelling performance from a limited collection of the annotated experimental data of a real-world practical thermography experiment.

scholarly journals Predicting Retinal Diseases using Efficient Image Processing and Convolutional Neural Network (CNN)

2021 ◽  
pp. 221-227
Author(s):  
Asif Mohammad ◽  
Mahruf Zaman Utso ◽  
Shifat Bin Habib ◽  
Amit Kumar Das
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Processing ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Retinal Images ◽  
Retinal Diseases ◽  
Medical Sector ◽  
The Neural Network ◽  
Computational Technology

Neural networks in image processing are becoming a more crucial and integral part of machine learning as computational technology and hardware systems are advanced. Deep learning is also getting attention from the medical sector as it is a prominent process for classifying diseases.  There is a lot of research to predict retinal diseases using deep learning algorithms like Convolutional Neural Network (CNN). Still, there are not many researches for predicting diseases like CNV which stands for choroidal neovascularization, DME, which stands for Diabetic Macular Edema; and DRUSEN. In our research paper, the CNN (Convolutional Neural Networks) algorithm labeled the dataset of OCT retinal images into four types: CNV, DME, DRUSEN, and Natural Retina. We have also done several preprocessing on the images before passing these to the neural network. We have implemented different models for our algorithm where individual models have different hidden layers.  At the end of our following research, we have found that our algorithm CNN generates 93% accuracy.

scholarly journals Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning

2019 ◽  
Author(s):  
Evgenii Tsymbalov ◽  
Sergei Makarychev ◽  
Alexander Shapeev ◽  
Maxim Panov
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Active Learning ◽  
Gaussian Processes ◽  
Large Scale ◽  
Training Sample ◽  
Real World Data ◽  
The Neural Network ◽  
Uncertainty Estimates ◽  
Out Of Sample

Active learning methods for neural networks are usually based on greedy criteria, which ultimately give a single new design point for the evaluation. Such an approach requires either some heuristics to sample a batch of design points at one active learning iteration, or retraining the neural network after adding each data point, which is computationally inefficient. Moreover, uncertainty estimates for neural networks sometimes are overconfident for the points lying far from the training sample. In this work, we propose to approximate Bayesian neural networks (BNN) by Gaussian processes (GP), which allows us to update the uncertainty estimates of predictions efficiently without retraining the neural network while avoiding overconfident uncertainty prediction for out-of-sample points. In a series of experiments on real-world data, including large-scale problems of chemical and physical modeling, we show the superiority of the proposed approach over the state-of-the-art methods.

Damage detection in a novel deep-learning framework: a robust method for feature extraction

2019 ◽  
Vol 19 (2) ◽  
pp. 424-442 ◽  
Author(s):  
Tian Guo ◽  
Lianping Wu ◽  
Cunjun Wang ◽  
Zili Xu
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Network Architecture ◽  
Mode Shapes ◽  
Damage Scenarios ◽  
Learning Modules ◽  
Learning Framework ◽  
Multi Scale ◽  
Extensive Evaluation ◽  
The Neural Network

Extracting damage features precisely while overcoming the adverse interferences of measurement noise and incomplete data is a problem demanding prompt solution in structural health monitoring (SHM). In this article, we present a deep-learning-based method that can extract the damage features from mode shapes without utilizing any hand-engineered feature or prior knowledge. To meet various requirements of the damage scenarios, we use convolutional neural network (CNN) algorithm and design a new network architecture: a multi-scale module, which helps in extracting features at various scales that can reduce the interference of contaminated data; stacked residual learning modules, which help in accelerating the network convergence; and a global average pooling layer, which helps in reducing the consumption of computing resources and obtaining a regression performance. An extensive evaluation of the proposed method is conducted by using datasets based on numerical simulations, along with two datasets based on laboratory measurements. The transferring parameter methodology is introduced to reduce retraining requirement without any decreases in precision. Furthermore, we plot the feature vectors of each layer to discuss the damage features learned at these layers and additionally provide the basis for explaining the working principle of the neural network. The results show that our proposed method has accuracy improvements of at least 10% over other network architectures.

scholarly journals Convolutional neural network for image classification based on transfer learning technique

2021 ◽  
Author(s):  
Ghassan Mohammed Halawani
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Image Classification ◽  
Transfer Learning ◽  
Learning Networks ◽  
The Neural Network ◽  
Learning Technique ◽  
Common Architecture

The main purpose of this project is to modify a convolutional neural network for image classification, based on a deep-learning framework. A transfer learning technique is used by the MATLAB interface to Alex-Net to train and modify the parameters in the last two fully connected layers of Alex-Net with a new dataset to perform classifications of thousands of images. First, the general common architecture of most neural networks and their benefits are presented. The mathematical models and the role of each part in the neural network are explained in detail. Second, different neural networks are studied in terms of architecture, application, and the working method to highlight the strengths and weaknesses of each of neural network. The final part conducts a detailed study on one of the most powerful deep-learning networks in image classification – i.e. the convolutional neural network – and how it can be modified to suit different classification tasks by using transfer learning technique in MATLAB.

Sign in / Sign up

Export Citation Format

Share Document