scholarly journals Artificial Neural Networks and Deep Learning for Genomic Prediction of Continuous Outcomes

2022 ◽  
pp. 427-476
Author(s):  
Osval Antonio Montesinos López ◽  
Abelardo Montesinos López ◽  
Jose Crossa
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Learning Rate ◽  
Continuous Outcomes ◽  
General Guide ◽  
Tuning Process ◽  
Hidden Layer ◽  
Learning Frameworks

AbstractThis chapter provides elements for implementing deep neural networks (deep learning) for continuous outcomes. We give details of the hyperparameters to be tuned in deep neural networks and provide a general guide for doing this task with more probability of success. Then we explain the most popular deep learning frameworks that can be used to implement these models as well as the most popular optimizers available in many software programs for deep learning. Several practical examples with plant breeding data for implementing deep neural networks in the Keras library are outlined. These examples take into account many components in the predictor as well many hyperparameters (hidden layer, number of neurons, learning rate, optimizers, penalization, etc.) for which we also illustrate how the tuning process can be done to increase the probability of a successful application.

SỬ DỤNG DEEP NEURAL NETWORKS PHÁT HIỆN GAI ĐỘNG KINH TRONG BẢN GHI EEG

2020 ◽  
pp. 77-84
Author(s):  
Xuyến
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Artificial Neural

Deep Neural Networks là một thuật toán dạy cho máy học, là phương pháp nâng cao của mạng nơ-ron nhân tạo (Artificial Neural Networks) nhiều tầng để học biểu diễn mô hình đối tượng. Bài báo trình bày phương pháp để phát hiện spike tự động, giải quyết bài toán cho các bác sỹ khi phân tích dữ liệu khổng lồ được thu thập từ bản ghi điện não để xác định một khu vực của não gây ra chứng động kinh. Hàng triệu mẫu được phân tích thủ công đã được đào tạo lại để tìm các gai liêp tiếp phát ra từ vùng não bị ảnh hưởng. Để đánh giá phương pháp đề xuất, tác giả đã xây dựng hệ thống trong đó sử dụng một số mô hình deep learning đưa vào thử nghiệm hỗ trợ các bác sỹ khám phát hiện và chẩn đoán sớm bệnh.

scholarly journals A new strategy for curriculum learning using model distillation

2020 ◽  
Vol 10 (2) ◽  
pp. 57-65
Author(s):  
Kaan Karakose ◽  
Metin Bilgin
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Mixed Data ◽  
Training Set ◽  
Complex Samples ◽  
New Strategy ◽  
Sample Sorting

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. Humans and animals learn much better when gradually presented in a meaningful order showing more concepts and complex samples rather than randomly presenting the information. The use of such training strategies in the context of artificial neural networks is called curriculum learning. In this study, a strategy was developed for curriculum learning. Using the CIFAR-10 and CIFAR-100 training sets, the last few layers of the pre-trained on ImageNet Xception model were trained to keep the training set knowledge in the model’s weight. Finally, a much smaller model was trained with the sample sorting methods presented using these difficulty levels. The findings obtained in this study show that the accuracy value generated when trained by the method we provided with the accuracy value trained with randomly mixed data was more than 1% for each epoch.   Keywords: Curriculum learning, model distillation, deep learning, academia, neural networks.

scholarly journals Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustness

2019 ◽  
Author(s):  
NhatHai Phan ◽  
Minh N. Vu ◽  
Yang Liu ◽  
Ruoming Jin ◽  
Dejing Dou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Theoretical Analysis ◽  
Gaussian Noise ◽  
Deep Neural Networks ◽  
Differential Privacy ◽  
Trade Off ◽  
Adversarial Examples ◽  
Hidden Layer ◽  
Privacy Budget

In this paper, we propose a novel Heterogeneous Gaussian Mechanism (HGM) to preserve differential privacy in deep neural networks, with provable robustness against adversarial examples. We first relax the constraint of the privacy budget in the traditional Gaussian Mechanism from (0, 1] to (0, infty), with a new bound of the noise scale to preserve differential privacy. The noise in our mechanism can be arbitrarily redistributed, offering a distinctive ability to address the trade-off between model utility and privacy loss. To derive provable robustness, our HGM is applied to inject Gaussian noise into the first hidden layer. Then, a tighter robustness bound is proposed. Theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of differentially private deep neural networks, compared with baseline approaches, under a variety of model attacks.

scholarly journals Analyzing and Visualizing Deep Neural Networks for Speech Recognition with Saliency-Adjusted Neuron Activation Profiles

Electronics ◽  
2021 ◽  
Vol 10 (11) ◽  
pp. 1350
Author(s):  
Andreas Krug ◽  
Maral Ebrahimzadeh ◽  
Jost Alemann ◽  
Jens Johannsmeier ◽  
Sebastian Stober
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Comparative Analysis ◽  
Speech Recognition ◽  
Automatic Speech Recognition ◽  
Deep Neural Networks ◽  
Neuron Activation ◽  
Flexible Framework

Deep Learning-based Automatic Speech Recognition (ASR) models are very successful, but hard to interpret. To gain a better understanding of how Artificial Neural Networks (ANNs) accomplish their tasks, several introspection methods have been proposed. However, established introspection techniques are mostly designed for computer vision tasks and rely on the data being visually interpretable, which limits their usefulness for understanding speech recognition models. To overcome this limitation, we developed a novel neuroscience-inspired technique for visualizing and understanding ANNs, called Saliency-Adjusted Neuron Activation Profiles (SNAPs). SNAPs are a flexible framework to analyze and visualize Deep Neural Networks that does not depend on visually interpretable data. In this work, we demonstrate how to utilize SNAPs for understanding fully-convolutional ASR models. This includes visualizing acoustic concepts learned by the model and the comparative analysis of their representations in the model layers.

scholarly journals Implementasi Jaringan Syaraf Tiruan Perambatan Balik untuk Memprediksi Harga Logam Mulia Emas Menggunakan Algoritma Lavenberg Marquardt

2013 ◽  
Vol 1 (2) ◽  
pp. 49
Author(s):  
Reza Najib Hidayat ◽  
R. Rizal Isnanto ◽  
Oky Dwi Nurhayati
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Fault Tolerance ◽  
Stock Market ◽  
Error Rate ◽  
Learning Rate ◽  
Training Data ◽  
Accuracy Rate ◽  
Rate Minimum ◽  
Hidden Layer

Gold is one of commodities investment which its value continue to increase by year. The rising price of gold will encourage investors to choose to invest in gold rather than the stock market. With the risks that are relatively low, gold can give better resultsin accordance with its increasing price. In addition, gold can also be a safe value protector in the future.The Objectives of the research are to predict the price of gold using artificial neural networks backpropagations methods and to analyze best network used in prediction. In the process of training data, it is used some training parameters to decide the best gold prediction architecture. Comparative parameters that is used are the variation of the number of hidden layers, number of neurons in each hidden layer, learning rate, minimum gradients and fault tolerance. The results showed that the best architecture has an accuracy rate of 99,7604% of data training and test data at 98,849% with architecture combinations are have two hidden layer neurons combined 10-30, the error rate 0.00001 and 0.00001 of learning rate.

scholarly journals Deep Learning Neural Networks Trained with MODIS Satellite-Derived Predictors for Long-Term Global Solar Radiation Prediction

Energies ◽  
2019 ◽  
Vol 12 (12) ◽  
pp. 2407 ◽  
Author(s):  
Ghimire ◽  
Deo ◽  
Raj ◽  
Mi
Keyword(s):  
Neural Networks ◽  
Feature Selection ◽  
Renewable Energy ◽  
Deep Learning ◽  
Solar Radiation ◽  
Solar Energy ◽  
Deep Neural Networks ◽  
Global Solar Radiation ◽  
Hidden Layer

Solar energy predictive models designed to emulate the long-term (e.g., monthly) global solar radiation (GSR) trained with satellite-derived predictors can be employed as decision tenets in the exploration, installation and management of solar energy production systems in remote and inaccessible solar-powered sites. In spite of a plethora of models designed for GSR prediction, deep learning, representing a state-of-the-art intelligent tool, remains an attractive approach for renewable energy exploration, monitoring and forecasting. In this paper, algorithms based on deep belief networks and deep neural networks are designed to predict long-term GSR. Deep learning algorithms trained with publicly-accessible Moderate Resolution Imaging Spectroradiometer (MODIS) satellite data are tested in Australia’s solar cities to predict the monthly GSR: single hidden layer and ensemble models. The monthly-scale MODIS-derived predictors (2003–2018) are adopted, with 15 diverse feature selection approaches including a Gaussian Emulation Machine for sensitivity analysis used to select optimal MODIS-predictor variables to simulate GSR against ground-truth values. Several statistical score metrics are adopted to comprehensively verify surface GSR simulations to ascertain the practicality of deep belief and deep neural networks. In the testing phase, deep learning models generate significantly lower absolute percentage bias (≤3%) and high Kling–Gupta efficiency (≥97.5%) values compared to the single hidden layer and ensemble model. This study ascertains that the optimal MODIS input variables employed in GSR prediction for solar energy applications can be relatively different for diverse sites, advocating a need for feature selection prior to the modelling of GSR. The proposed deep learning approach can be adopted to identify solar energy potential proactively in locations where it is impossible to install an environmental monitoring data acquisition instrument. Hence, MODIS and other related satellite-derived predictors can be incorporated for solar energy prediction as a strategy for long-term renewable energy exploration.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

scholarly journals Automatic Detection of Arrhythmia Based on Multi-Resolution Representation of ECG Signal

Sensors ◽  
2020 ◽  
Vol 20 (6) ◽  
pp. 1579
Author(s):  
Dongqi Wang ◽  
Qinghua Meng ◽  
Dongming Chen ◽  
Hupo Zhang ◽  
Lisheng Xu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Channel Model ◽  
Expert Knowledge ◽  
Automatic Detection ◽  
Data Representation ◽  
Learning Technology ◽  
Arrhythmia Detection ◽  
Automatic Feature Extraction

Automatic detection of arrhythmia is of great significance for early prevention and diagnosis of cardiovascular disease. Traditional feature engineering methods based on expert knowledge lack multidimensional and multi-view information abstraction and data representation ability, so the traditional research on pattern recognition of arrhythmia detection cannot achieve satisfactory results. Recently, with the increase of deep learning technology, automatic feature extraction of ECG data based on deep neural networks has been widely discussed. In order to utilize the complementary strength between different schemes, in this paper, we propose an arrhythmia detection method based on the multi-resolution representation (MRR) of ECG signals. This method utilizes four different up to date deep neural networks as four channel models for ECG vector representations learning. The deep learning based representations, together with hand-crafted features of ECG, forms the MRR, which is the input of the downstream classification strategy. The experimental results of big ECG dataset multi-label classification confirm that the F1 score of the proposed method is 0.9238, which is 1.31%, 0.62%, 1.18% and 0.6% higher than that of each channel model. From the perspective of architecture, this proposed method is highly scalable and can be employed as an example for arrhythmia recognition.

scholarly journals Enabling deeper learning on big data for materials informatics applications

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Dipendra Jha ◽  
Vishu Gupta ◽  
Logan Ward ◽  
Zijiang Yang ◽  
Christopher Wolverton ◽  
...  
Keyword(s):  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Materials Science ◽  
Prediction Models ◽  
Model Performance ◽  
Materials Informatics ◽  
Learning Framework ◽  
Significant Attention

AbstractThe application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

scholarly journals Representing Deep Neural Networks Latent Space Geometries with Graphs

Algorithms ◽  
2021 ◽  
Vol 14 (2) ◽  
pp. 39
Author(s):  
Carlos Lassance ◽  
Vincent Gripon ◽  
Antonio Ortega
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Learning ◽  
Objective Function ◽  
Learning Process ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
The Core ◽  
Learning Tasks ◽  
Latent Space

Deep Learning (DL) has attracted a lot of attention for its ability to reach state-of-the-art performance in many machine learning tasks. The core principle of DL methods consists of training composite architectures in an end-to-end fashion, where inputs are associated with outputs trained to optimize an objective function. Because of their compositional nature, DL architectures naturally exhibit several intermediate representations of the inputs, which belong to so-called latent spaces. When treated individually, these intermediate representations are most of the time unconstrained during the learning process, as it is unclear which properties should be favored. However, when processing a batch of inputs concurrently, the corresponding set of intermediate representations exhibit relations (what we call a geometry) on which desired properties can be sought. In this work, we show that it is possible to introduce constraints on these latent geometries to address various problems. In more detail, we propose to represent geometries by constructing similarity graphs from the intermediate representations obtained when processing a batch of inputs. By constraining these Latent Geometry Graphs (LGGs), we address the three following problems: (i) reproducing the behavior of a teacher architecture is achieved by mimicking its geometry, (ii) designing efficient embeddings for classification is achieved by targeting specific geometries, and (iii) robustness to deviations on inputs is achieved via enforcing smooth variation of geometry between consecutive latent spaces. Using standard vision benchmarks, we demonstrate the ability of the proposed geometry-based methods in solving the considered problems.

Sign in / Sign up

Export Citation Format

Share Document