Uncertainty Quantification and Optimization of Deep Learning for Fracture Recognition

2021 ◽  
Author(s):  
Ryan Santoso ◽  
Xupeng He ◽  
Marwa Alsinan ◽  
Hyung Kwak ◽  
Hussein Hoteit
Keyword(s):  
Deep Learning ◽  
Uncertainty Quantification ◽  
Learning Rate ◽  
Batch Size ◽  
Global Maximum ◽  
Fractured Reservoir ◽  
Training Set ◽  
Fast Learning ◽  
Learning Rates ◽  
Aleatoric Uncertainty

Abstract Automatic fracture recognition from borehole images or outcrops is applicable for the construction of fractured reservoir models. Deep learning for fracture recognition is subject to uncertainty due to sparse and imbalanced training set, and random initialization. We present a new workflow to optimize a deep learning model under uncertainty using U-Net. We consider both epistemic and aleatoric uncertainty of the model. We propose a U-Net architecture by inserting dropout layer after every "weighting" layer. We vary the dropout probability to investigate its impact on the uncertainty response. We build the training set and assign uniform distribution for each training parameter, such as the number of epochs, batch size, and learning rate. We then perform uncertainty quantification by running the model multiple times for each realization, where we capture the aleatoric response. In this approach, which is based on Monte Carlo Dropout, the variance map and F1-scores are utilized to evaluate the need to craft additional augmentations or stop the process. This work demonstrates the existence of uncertainty within the deep learning caused by sparse and imbalanced training sets. This issue leads to unstable predictions. The overall responses are accommodated in the form of aleatoric uncertainty. Our workflow utilizes the uncertainty response (variance map) as a measure to craft additional augmentations in the training set. High variance in certain features denotes the need to add new augmented images containing the features, either through affine transformation (rotation, translation, and scaling) or utilizing similar images. The augmentation improves the accuracy of the prediction, reduces the variance prediction, and stabilizes the output. Architecture, number of epochs, batch size, and learning rate are optimized under a fixed-uncertain training set. We perform the optimization by searching the global maximum of accuracy after running multiple realizations. Besides the quality of the training set, the learning rate is the heavy-hitter in the optimization process. The selected learning rate controls the diffusion of information in the model. Under the imbalanced condition, fast learning rates cause the model to miss the main features. The other challenge in fracture recognition on a real outcrop is to optimally pick the parental images to generate the initial training set. We suggest picking images from multiple sides of the outcrop, which shows significant variations of the features. This technique is needed to avoid long iteration within the workflow. We introduce a new approach to address the uncertainties associated with the training process and with the physical problem. The proposed approach is general in concept and can be applied to various deep-learning problems in geoscience.

scholarly journals A Zeroth-Order Adaptive Learning Rate Method to Reduce Cost of Hyperparameter Tuning for Deep Learning

2021 ◽  
Vol 11 (21) ◽  
pp. 10184
Author(s):  
Yanan Li ◽  
Xuebin Ren ◽  
Fangyuan Zhao ◽  
Shusen Yang
Keyword(s):  
Deep Learning ◽  
Network Architecture ◽  
Adaptive Methods ◽  
Learning Rate ◽  
Fine Tuning ◽  
Batch Size ◽  
Zeroth Order ◽  
Learning Rates ◽  
Initial Learning ◽  
Rate Method

Due to powerful data representation ability, deep learning has dramatically improved the state-of-the-art in many practical applications. However, the utility highly depends on fine-tuning of hyper-parameters, including learning rate, batch size, and network initialization. Although many first-order adaptive methods (e.g., Adam, Adagrad) have been proposed to adjust learning rate based on gradients, they are susceptible to the initial learning rate and network architecture. Therefore, the main challenge of using deep learning in practice is how to reduce the cost of tuning hyper-parameters. To address this, we propose a heuristic zeroth-order learning rate method, Adacomp, which adaptively adjusts the learning rate based only on values of the loss function. The main idea is that Adacomp penalizes large learning rates to ensure the convergence and compensates small learning rates to accelerate the training process. Therefore, Adacomp is robust to the initial learning rate. Extensive experiments, including comparison to six typically adaptive methods (Momentum, Adagrad, RMSprop, Adadelta, Adam, and Adamax) on several benchmark datasets for image classification tasks (MNIST, KMNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100), were conducted. Experimental results show that Adacomp is not only robust to the initial learning rate but also to the network architecture, network initialization, and batch size.

scholarly journals Segmentasi Tumor Otak Berdasarkan Citra Magnetic Resonance Imaging Dengan Menggunakan Metode U-NET

2020 ◽  
Vol 19 (2) ◽  
pp. 151
Author(s):  
Ida Bagus Leo Mahadya Suta ◽  
Made Sudarma ◽  
I Nyoman Satya Kumara
Keyword(s):  
Magnetic Resonance Imaging ◽  
Deep Learning ◽  
Magnetic Resonance ◽  
Learning Rate ◽  
Batch Size ◽  
Resonance Imaging ◽  
Dice Coefficient ◽  
Magnetic Resonance Imaging Mri

Tumor otak merupakan salah satu penyakit yang mematikan dimana 3.7% per 100.000 pasien mengidap tumor ganas. Untuk menganalisa tumor otak dapat dilakukan melalui segmentasi citra Magnetic Resonance Imaging (MRI). Proses analisa citra secara otomatis dibutuhkan untuk menghemat waktu dan meningkatkan akurasi dari diagnosa yang dilakukan. Segmentasi secara otomatis dapat dilakukan dengan deep learning. U-NET merupakan salah satu metode yang digunakan untuk melakukan segmentasi citra medis karena bekerja dapa pixel level. Dengan menerapkan fungsi aktivasi ReLU dan Adam Optimizer, metode ini dapat menyelesaikan permasalahan segmentasi tumor otak. Dataset untuk proses training dan validation menggunakan BRATS 2017. Beberapa hyperparameter diterapkan pada metode ini yaitu, learning rate (lr) = 0.0001, batch size (bz) = 5, epoch = 80 dan beta (  ) = 0.9. Dari serangkaian proses yang dilakukan, akurasi metode U-NET dihitung dengan rumus Dice Coefficient dan menghasilkan nilai akurasi sebagai berikut: 90.22% (Full Tumor), 78.09% (Core Tumor) dan 80.20% (Enhancing Tumor).

scholarly journals Optimal Accuracy Zone Identification in Object Detection Technique - A Learning Rate Methodology

2019 ◽  
Vol 9 (1) ◽  
pp. 6470-6476
Keyword(s):  
Deep Learning ◽  
Object Detection ◽  
Learning Rate ◽  
Training Data ◽  
Fine Tuning ◽  
Batch Size ◽  
Detection Accuracy ◽  
Learning Models ◽  
The Impact ◽  
Selection Of

In the recent past, Deep Learning models [1] are predominantly being used in Object Detection algorithms due to their accurate Image Recognition capability. These models extract features from the input images and videos [2] for identification of objects present in them. Various applications of these models include Image Processing, Video analysis, Speech Recognition, Biomedical Image Analysis, Biometric Recognition, Iris Recognition, National Security applications, Cyber Security, Natural Language Processing [3], Weather Forecasting applications, Renewable Energy Generation Scheduling etc. These models utilize the concept of Convolutional Neural Network (CNN) [3], which constitutes several layers of artificial neurons. The accuracy of Deep Learning models [1] depends on various parameters such as ‘Learning-rate’, ‘Training batch size’, ‘Validation batch size’, ‘Activation Function’, ‘Drop-out rate’ etc. These parameters are known as Hyper-Parameters. Object detection accuracy depends on selection of Hyperparameters and these in-turn decides the optimum accuracy. Hence, finding the best values for these parameters is a challenging task. Fine-Tuning is a process used for selection of a suitable Hyper-Parameter value for improvement of object detection accuracy. Selection of an inappropriate Hyper-Parameter value, leads to Over-Fitting or Under-Fitting of data. Over-Fitting is a case, when training data is larger than the required, which results in learning noise and inaccurate object detection. Under-fitting is a case, when the model is unable to capture the trend of the data and which leads to more erroneous results in testing or training data. In this paper, a balance between Over-fitting and Under-fitting is achieved by varying the ‘Learning rate’ of various Deep Learning models. Four Deep Learning Models such as VGG16, VGG19, InceptionV3 and Xception are considered in this paper for analysis purpose. The best zone of Learning-rate for each model, in respect of maximum Object Detection accuracy, is analyzed. In this paper a dataset of 70 object classes is taken and the prediction accuracy is analyzed by changing the ‘Learning-rate’ and keeping the rest of the Hyper-Parameters constant. This paper mainly concentrates on the impact of ‘Learning-rate’ on accuracy and identifies an optimum accuracy zone in Object Detection

scholarly journals Towards Detecting Dementia via Deep Learning

2021 ◽  
Vol 16 (4) ◽  
pp. 1-17
Author(s):  
Deepika Bansal ◽  
*Kavita Khanna ◽  
Rita Chhikara ◽  
Rakesh Kumar Dua ◽  
Rajeev Malini
Keyword(s):  
Deep Learning ◽  
Learning Rate ◽  
Batch Size ◽  
Brain Disorder ◽  
Convolutional Network ◽  
Diagnosis Of Dementia ◽  
Multi Class Classification ◽  
Classifier Learning ◽  
Best Parameters ◽  
Global Health Problem

Dementia is a brain disorder that causes loss of memory leading to disruption in the normal course of life of an individual. It is emerging as a global health problem in adults with age 65 years or above. Early diagnosis of dementia has gone forth as a key research zone with the aim of early identification for hindering the advancement. Deep learning provides path-breaking applications in medical imaging. This study provides a detailed summary of different implementation approaches of deep learning for detecting the disease. Transfer learning for multi-class classification has also been explored for detecting dementia. The pre-trained convolutional network, AlexNet is used with 3 optimizers, SGDM, ADAM, RMSProp. A Dataset of 60 MRI images is taken from the OASIS dataset. Accuracy of the methods has been compared and the best parameters including classifier, learning rate, and a batch size of the model have been identified. SGDM classifier with a learning rate 10-4 and a mini-batch size of 10 have shown the best performance in a reasonable time.

scholarly journals Learning Rate Optimization in CNN for Accurate Ophthalmic Classification

2021 ◽  
Vol 10 (4) ◽  
pp. 211-216
Author(s):  
Mahmoud Smaida ◽  
Serhii Yaroshchak ◽  
Ahmed Y. Ben Sasi
Keyword(s):  
Deep Learning ◽  
Adaptive Learning ◽  
Optimization Techniques ◽  
Learning Rate ◽  
Training Set ◽  
Model Accuracy ◽  
Adaptive Learning Rate ◽  
Rate Method ◽  
Model Training ◽  
Rate Optimization

One of the most important hyper-parameters for model training and generalization is the learning rate. Recently, many research studies have shown that optimizing the learning rate schedule is very useful for training deep neural networks to get accurate and efficient results. In this paper, different learning rate schedules using some comprehensive optimization techniques have been compared in order to measure the accuracy of a convolutional neural network CNN model to classify four ophthalmic conditions. In this work, a deep learning CNN based on Keras and TensorFlow has been deployed using Python on a database that contains 1692 images, which consists of four types of ophthalmic cases: Glaucoma, Myopia, Diabetic retinopathy, and Normal eyes. The CNN model has been trained on Google Colab. GPU with different learning rate schedules and adaptive learning algorithms. Constant learning rate, time-based decay, step-based decay, exponential decay, and adaptive learning rate optimization techniques for deep learning have been addressed. Adam adaptive learning rate method. has outperformed the other optimization techniques and achieved the best model accuracy of 92.58% for training set and 80.49% for validation datasets, respectively.

scholarly journals Real UAV-Bird Image Classification Using CNN with a Synthetic Dataset

2021 ◽  
Vol 11 (9) ◽  
pp. 3863
Author(s):  
Ali Emre Öztürk ◽  
Ergun Erçelebi
Keyword(s):  
Deep Learning ◽  
Image Classification ◽  
Synthetic Data ◽  
Real Data ◽  
Corner Detection ◽  
Batch Size ◽  
Test Accuracy ◽  
Classification Problems ◽  
Auc Value ◽  
Classification Test

A large amount of training image data is required for solving image classification problems using deep learning (DL) networks. In this study, we aimed to train DL networks with synthetic images generated by using a game engine and determine the effects of the networks on performance when solving real-image classification problems. The study presents the results of using corner detection and nearest three-point selection (CDNTS) layers to classify bird and rotary-wing unmanned aerial vehicle (RW-UAV) images, provides a comprehensive comparison of two different experimental setups, and emphasizes the significant improvements in the performance in deep learning-based networks due to the inclusion of a CDNTS layer. Experiment 1 corresponds to training the commonly used deep learning-based networks with synthetic data and an image classification test on real data. Experiment 2 corresponds to training the CDNTS layer and commonly used deep learning-based networks with synthetic data and an image classification test on real data. In experiment 1, the best area under the curve (AUC) value for the image classification test accuracy was measured as 72%. In experiment 2, using the CDNTS layer, the AUC value for the image classification test accuracy was measured as 88.9%. A total of 432 different combinations of trainings were investigated in the experimental setups. The experiments were trained with various DL networks using four different optimizers by considering all combinations of batch size, learning rate, and dropout hyperparameters. The test accuracy AUC values for networks in experiment 1 ranged from 55% to 74%, whereas the test accuracy AUC values in experiment 2 networks with a CDNTS layer ranged from 76% to 89.9%. It was observed that the CDNTS layer has considerable effects on the image classification accuracy performance of deep learning-based networks. AUC, F-score, and test accuracy measures were used to validate the success of the networks.

scholarly journals Simba

2021 ◽  
Vol 64 (6) ◽  
pp. 107-116
Author(s):  
Yakun Sophia Shao ◽  
Jason Cemons ◽  
Rangharajan Venkatesan ◽  
Brian Zimmer ◽  
Matthew Fojtik ◽  
...  
Keyword(s):  
Deep Learning ◽  
Large Scale ◽  
Data Locality ◽  
Coarse Grained ◽  
Batch Size ◽  
Peak Performance ◽  
Large Scale Systems ◽  
High Area ◽  
On Chip ◽  
And Storage

Package-level integration using multi-chip-modules (MCMs) is a promising approach for building large-scale systems. Compared to a large monolithic die, an MCM combines many smaller chiplets into a larger system, substantially reducing fabrication and design costs. Current MCMs typically only contain a handful of coarse-grained large chiplets due to the high area, performance, and energy overheads associated with inter-chiplet communication. This work investigates and quantifies the costs and benefits of using MCMs with finegrained chiplets for deep learning inference, an application domain with large compute and on-chip storage requirements. To evaluate the approach, we architected, implemented, fabricated, and tested Simba, a 36-chiplet prototype MCM system for deep-learning inference. Each chiplet achieves 4 TOPS peak performance, and the 36-chiplet MCM package achieves up to 128 TOPS and up to 6.1 TOPS/W. The MCM is configurable to support a flexible mapping of DNN layers to the distributed compute and storage units. To mitigate inter-chiplet communication overheads, we introduce three tiling optimizations that improve data locality. These optimizations achieve up to 16% speedup compared to the baseline layer mapping. Our evaluation shows that Simba can process 1988 images/s running ResNet-50 with a batch size of one, delivering an inference latency of 0.50 ms.

scholarly journals Image-based textile decoding

2020 ◽  
pp. 1-14
Author(s):  
Siqiang Chen ◽  
Masahiro Toyoura ◽  
Takamasa Terada ◽  
Xiaoyang Mao ◽  
Gang Xu
Keyword(s):  
Deep Learning ◽  
Grid Point ◽  
Intermediate Representation ◽  
Training Set ◽  
Binary Matrix ◽  
The Matrix ◽  
Correct Pattern ◽  
Grid Points ◽  
Textile Fabric ◽  
Direct Correspondence

A textile fabric consists of countless parallel vertical yarns (warps) and horizontal yarns (wefts). While common looms can weave repetitive patterns, Jacquard looms can weave the patterns without repetition restrictions. A pattern in which the warps and wefts cross on a grid is defined in a binary matrix. The binary matrix can define which warp and weft is on top at each grid point of the Jacquard fabric. The process can be regarded as encoding from pattern to textile. In this work, we propose a decoding method that generates a binary pattern from a textile fabric that has been already woven. We could not use a deep neural network to learn the process based solely on the training set of patterns and observed fabric images. The crossing points in the observed image were not completely located on the grid points, so it was difficult to take a direct correspondence between the fabric images and the pattern represented by the matrix in the framework of deep learning. Therefore, we propose a method that can apply the framework of deep learning viau the intermediate representation of patterns and images. We show how to convert a pattern into an intermediate representation and how to reconvert the output into a pattern and confirm its effectiveness. In this experiment, we confirmed that 93% of correct pattern was obtained by decoding the pattern from the actual fabric images and weaving them again.

Uncertainty Quantification in Skin Cancer Classification using Three-Way Decision-based Bayesian Deep Learning

2021 ◽  
pp. 104418
Author(s):  
Moloud Abdar ◽  
Maryam Samami ◽  
Sajjad Dehghani Mahmoodabad ◽  
Thang Doan ◽  
Bogdan Mazoure ◽  
...  
Keyword(s):  
Deep Learning ◽  
Skin Cancer ◽  
Uncertainty Quantification ◽  
Cancer Classification

Deep learning for denoising

Geophysics ◽  
2019 ◽  
Vol 84 (6) ◽  
pp. V333-V350 ◽  
Author(s):  
Siwei Yu ◽  
Jianwei Ma ◽  
Wenlong Wang
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Graphics Processing Unit ◽  
Random Noise ◽  
Stochastic Gradient Descent ◽  
Processing Unit ◽  
Noise Attenuation ◽  
Optimal Parameters ◽  
Training Set ◽  
Unknown Variance

Compared with traditional seismic noise attenuation algorithms that depend on signal models and their corresponding prior assumptions, removing noise with a deep neural network is trained based on a large training set in which the inputs are the raw data sets and the corresponding outputs are the desired clean data. After the completion of training, the deep-learning (DL) method achieves adaptive denoising with no requirements of (1) accurate modelings of the signal and noise or (2) optimal parameters tuning. We call this intelligent denoising. We have used a convolutional neural network (CNN) as the basic tool for DL. In random and linear noise attenuation, the training set is generated with artificially added noise. In the multiple attenuation step, the training set is generated with the acoustic wave equation. The stochastic gradient descent is used to solve the optimal parameters for the CNN. The runtime of DL on a graphics processing unit for denoising has the same order as the [Formula: see text]-[Formula: see text] deconvolution method. Synthetic and field results indicate the potential applications of DL in automatic attenuation of random noise (with unknown variance), linear noise, and multiples.

Sign in / Sign up

Export Citation Format

Share Document