scholarly journals Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality

2021 ◽  
Vol 7 (1) ◽  
Author(s):  
Rama K. Vasudevan ◽  
Maxim Ziatdinov ◽  
Lukas Vlcek ◽  
Sergei V. Kalinin
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Deep Learning ◽  
Bayesian Methods ◽  
Deep Neural Networks ◽  
Applied Research ◽  
Modern Science ◽  
Generative Models ◽  
Knowledge Development ◽  
Physical Constraints

AbstractDeep neural networks (‘deep learning’) have emerged as a technology of choice to tackle problems in speech recognition, computer vision, finance, etc. However, adoption of deep learning in physical domains brings substantial challenges stemming from the correlative nature of deep learning methods compared to the causal, hypothesis driven nature of modern science. We argue that the broad adoption of Bayesian methods incorporating prior knowledge, development of solutions with incorporated physical constraints and parsimonious structural descriptors and generative models, and ultimately adoption of causal models, offers a path forward for fundamental and applied research.

scholarly journals Multimodal Multi-tasking for Skin Lesion Classification Using Deep Neural Networks

2021 ◽  
pp. 27-38
Author(s):  
Rafaela Carvalho ◽  
João Pedrosa ◽  
Tudor Nedelcu
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Deep Learning ◽  
Skin Lesion ◽  
Deep Neural Networks ◽  
Classification Performance ◽  
Additional Information ◽  
Deep Learning Model ◽  
Lesion Classification

AbstractSkin cancer is one of the most common types of cancer and, with its increasing incidence, accurate early diagnosis is crucial to improve prognosis of patients. In the process of visual inspection, dermatologists follow specific dermoscopic algorithms and identify important features to provide a diagnosis. This process can be automated as such characteristics can be extracted by computer vision techniques. Although deep neural networks can extract useful features from digital images for skin lesion classification, performance can be improved by providing additional information. The extracted pseudo-features can be used as input (multimodal) or output (multi-tasking) to train a robust deep learning model. This work investigates the multimodal and multi-tasking techniques for more efficient training, given the single optimization of several related tasks in the latter, and generation of better diagnosis predictions. Additionally, the role of lesion segmentation is also studied. Results show that multi-tasking improves learning of beneficial features which lead to better predictions, and pseudo-features inspired by the ABCD rule provide readily available helpful information about the skin lesion.

scholarly journals A new strategy for curriculum learning using model distillation

2020 ◽  
Vol 10 (2) ◽  
pp. 57-65
Author(s):  
Kaan Karakose ◽  
Metin Bilgin
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Mixed Data ◽  
Training Set ◽  
Complex Samples ◽  
New Strategy ◽  
Sample Sorting

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. Humans and animals learn much better when gradually presented in a meaningful order showing more concepts and complex samples rather than randomly presenting the information. The use of such training strategies in the context of artificial neural networks is called curriculum learning. In this study, a strategy was developed for curriculum learning. Using the CIFAR-10 and CIFAR-100 training sets, the last few layers of the pre-trained on ImageNet Xception model were trained to keep the training set knowledge in the model’s weight. Finally, a much smaller model was trained with the sample sorting methods presented using these difficulty levels. The findings obtained in this study show that the accuracy value generated when trained by the method we provided with the accuracy value trained with randomly mixed data was more than 1% for each epoch.   Keywords: Curriculum learning, model distillation, deep learning, academia, neural networks.

scholarly journals Analyzing and Visualizing Deep Neural Networks for Speech Recognition with Saliency-Adjusted Neuron Activation Profiles

Electronics ◽  
2021 ◽  
Vol 10 (11) ◽  
pp. 1350
Author(s):  
Andreas Krug ◽  
Maral Ebrahimzadeh ◽  
Jost Alemann ◽  
Jens Johannsmeier ◽  
Sebastian Stober
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Artificial Neural Networks ◽  
Deep Learning ◽  
Comparative Analysis ◽  
Speech Recognition ◽  
Automatic Speech Recognition ◽  
Deep Neural Networks ◽  
Neuron Activation ◽  
Flexible Framework

Deep Learning-based Automatic Speech Recognition (ASR) models are very successful, but hard to interpret. To gain a better understanding of how Artificial Neural Networks (ANNs) accomplish their tasks, several introspection methods have been proposed. However, established introspection techniques are mostly designed for computer vision tasks and rely on the data being visually interpretable, which limits their usefulness for understanding speech recognition models. To overcome this limitation, we developed a novel neuroscience-inspired technique for visualizing and understanding ANNs, called Saliency-Adjusted Neuron Activation Profiles (SNAPs). SNAPs are a flexible framework to analyze and visualize Deep Neural Networks that does not depend on visually interpretable data. In this work, we demonstrate how to utilize SNAPs for understanding fully-convolutional ASR models. This includes visualizing acoustic concepts learned by the model and the comparative analysis of their representations in the model layers.

scholarly journals Automatic Detection of Arrhythmia Based on Multi-Resolution Representation of ECG Signal

Sensors ◽  
2020 ◽  
Vol 20 (6) ◽  
pp. 1579
Author(s):  
Dongqi Wang ◽  
Qinghua Meng ◽  
Dongming Chen ◽  
Hupo Zhang ◽  
Lisheng Xu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Channel Model ◽  
Expert Knowledge ◽  
Automatic Detection ◽  
Data Representation ◽  
Learning Technology ◽  
Arrhythmia Detection ◽  
Automatic Feature Extraction

Automatic detection of arrhythmia is of great significance for early prevention and diagnosis of cardiovascular disease. Traditional feature engineering methods based on expert knowledge lack multidimensional and multi-view information abstraction and data representation ability, so the traditional research on pattern recognition of arrhythmia detection cannot achieve satisfactory results. Recently, with the increase of deep learning technology, automatic feature extraction of ECG data based on deep neural networks has been widely discussed. In order to utilize the complementary strength between different schemes, in this paper, we propose an arrhythmia detection method based on the multi-resolution representation (MRR) of ECG signals. This method utilizes four different up to date deep neural networks as four channel models for ECG vector representations learning. The deep learning based representations, together with hand-crafted features of ECG, forms the MRR, which is the input of the downstream classification strategy. The experimental results of big ECG dataset multi-label classification confirm that the F1 score of the proposed method is 0.9238, which is 1.31%, 0.62%, 1.18% and 0.6% higher than that of each channel model. From the perspective of architecture, this proposed method is highly scalable and can be employed as an example for arrhythmia recognition.

scholarly journals Enabling deeper learning on big data for materials informatics applications

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Dipendra Jha ◽  
Vishu Gupta ◽  
Logan Ward ◽  
Zijiang Yang ◽  
Christopher Wolverton ◽  
...  
Keyword(s):  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Materials Science ◽  
Prediction Models ◽  
Model Performance ◽  
Materials Informatics ◽  
Learning Framework ◽  
Significant Attention

AbstractThe application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

scholarly journals Deep neural networks for quantum circuit mapping

2021 ◽  
Author(s):  
Giovanni Acampora ◽  
Roberto Schiattarella
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Quantum Algorithms ◽  
Quantum Algorithm ◽  
Quantum Circuit ◽  
Machine Learning Techniques ◽  
Quantum Computers ◽  
Physical Constraints ◽  
Proper Mapping ◽  
Correct Execution

AbstractQuantum computers have become reality thanks to the effort of some majors in developing innovative technologies that enable the usage of quantum effects in computation, so as to pave the way towards the design of efficient quantum algorithms to use in different applications domains, from finance and chemistry to artificial and computational intelligence. However, there are still some technological limitations that do not allow a correct design of quantum algorithms, compromising the achievement of the so-called quantum advantage. Specifically, a major limitation in the design of a quantum algorithm is related to its proper mapping to a specific quantum processor so that the underlying physical constraints are satisfied. This hard problem, known as circuit mapping, is a critical task to face in quantum world, and it needs to be efficiently addressed to allow quantum computers to work correctly and productively. In order to bridge above gap, this paper introduces a very first circuit mapping approach based on deep neural networks, which opens a completely new scenario in which the correct execution of quantum algorithms is supported by classical machine learning techniques. As shown in experimental section, the proposed approach speeds up current state-of-the-art mapping algorithms when used on 5-qubits IBM Q processors, maintaining suitable mapping accuracy.

scholarly journals Representing Deep Neural Networks Latent Space Geometries with Graphs

Algorithms ◽  
2021 ◽  
Vol 14 (2) ◽  
pp. 39
Author(s):  
Carlos Lassance ◽  
Vincent Gripon ◽  
Antonio Ortega
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Learning ◽  
Objective Function ◽  
Learning Process ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
The Core ◽  
Learning Tasks ◽  
Latent Space

Deep Learning (DL) has attracted a lot of attention for its ability to reach state-of-the-art performance in many machine learning tasks. The core principle of DL methods consists of training composite architectures in an end-to-end fashion, where inputs are associated with outputs trained to optimize an objective function. Because of their compositional nature, DL architectures naturally exhibit several intermediate representations of the inputs, which belong to so-called latent spaces. When treated individually, these intermediate representations are most of the time unconstrained during the learning process, as it is unclear which properties should be favored. However, when processing a batch of inputs concurrently, the corresponding set of intermediate representations exhibit relations (what we call a geometry) on which desired properties can be sought. In this work, we show that it is possible to introduce constraints on these latent geometries to address various problems. In more detail, we propose to represent geometries by constructing similarity graphs from the intermediate representations obtained when processing a batch of inputs. By constraining these Latent Geometry Graphs (LGGs), we address the three following problems: (i) reproducing the behavior of a teacher architecture is achieved by mimicking its geometry, (ii) designing efficient embeddings for classification is achieved by targeting specific geometries, and (iii) robustness to deviations on inputs is achieved via enforcing smooth variation of geometry between consecutive latent spaces. Using standard vision benchmarks, we demonstrate the ability of the proposed geometry-based methods in solving the considered problems.

scholarly journals Deep neural networks for waves assisted by the Wiener–Hopf method

2020 ◽  
Vol 476 (2235) ◽  
pp. 20190846 ◽  
Author(s):  
Xun Huang
Keyword(s):  
Neural Networks ◽  
Wave Equations ◽  
Deep Neural Networks ◽  
Research Strategy ◽  
Propagation Model ◽  
Physical Constraints ◽  
Aerospace Applications ◽  
Feed Forward Network ◽  
New Perspective ◽  
Hopf Method

In this work, the classical Wiener–Hopf method is incorporated into the emerging deep neural networks for the study of certain wave problems. The essential idea is to use the first-principle-based analytical method to efficiently produce a large volume of datasets that would supervise the learning of data-hungry deep neural networks, and to further explain the working mechanisms on underneath. To demonstrate such a combinational research strategy, a deep feed-forward network is first used to approximate the forward propagation model of a duct acoustic problem, which can find important aerospace applications in aeroengine noise tests. Next, a convolutional type U-net is developed to learn spatial derivatives in wave equations, which could help to promote computational paradigm in mathematical physics and engineering applications. A couple of extensions of the U-net architecture are proposed to further impose possible physical constraints. Finally, after giving the implementation details, the performance of the neural networks are studied by comparing with analytical solutions from the Wiener–Hopf method. Overall, the Wiener–Hopf method is used here from a totally new perspective and such a combinational research strategy shall represent the key achievement of this work.

scholarly journals Diversity oriented Deep Reinforcement Learning for targeted molecule generation

2021 ◽  
Vol 13 (1) ◽  
Author(s):  
Tiago Pereira ◽  
Maryam Abbasi ◽  
Bernardete Ribeiro ◽  
Joel P. Arrais
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Reinforcement Learning ◽  
Deep Neural Networks ◽  
Chemical Space ◽  
Biological Properties ◽  
Training Process ◽  
Training Strategy ◽  
Inhibitory Power ◽  
Exploratory Strategy

AbstractIn this work, we explore the potential of deep learning to streamline the process of identifying new potential drugs through the computational generation of molecules with interesting biological properties. Two deep neural networks compose our targeted generation framework: the Generator, which is trained to learn the building rules of valid molecules employing SMILES strings notation, and the Predictor which evaluates the newly generated compounds by predicting their affinity for the desired target. Then, the Generator is optimized through Reinforcement Learning to produce molecules with bespoken properties. The innovation of this approach is the exploratory strategy applied during the reinforcement training process that seeks to add novelty to the generated compounds. This training strategy employs two Generators interchangeably to sample new SMILES: the initially trained model that will remain fixed and a copy of the previous one that will be updated during the training to uncover the most promising molecules. The evolution of the reward assigned by the Predictor determines how often each one is employed to select the next token of the molecule. This strategy establishes a compromise between the need to acquire more information about the chemical space and the need to sample new molecules, with the experience gained so far. To demonstrate the effectiveness of the method, the Generator is trained to design molecules with an optimized coefficient of partition and also high inhibitory power against the Adenosine $$A_{2A}$$ A 2 A and $$\kappa$$ κ opioid receptors. The results reveal that the model can effectively adjust the newly generated molecules towards the wanted direction. More importantly, it was possible to find promising sets of unique and diverse molecules, which was the main purpose of the newly implemented strategy.

Sign in / Sign up

Export Citation Format

Share Document