An Overview of Deep Learning Architecture of Deep Neural Networks and Autoencoders

2020 ◽  
Vol 17 (1) ◽  
pp. 182-188 ◽  
Author(s):  
Mohit Sewak ◽  
Sanjay K. Sahay ◽  
Hemant Rathore
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Supervised Learning ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Decision Criteria ◽  
Architectural Features ◽  
Size Number ◽  
Selection Decisions ◽  
Great Progress

The recent wide applications of deep learning in multiple fields has shown a great progress, but to perform optimally, it requires the adjustment of various architectural features and hyper-parameters. Moreover, deep learning could be used with multiple varieties of architecture aimed at different objectives, e.g., autoencoders are popular for un-supervised learning applications for reducing the dimensionality of the dataset. Similarly, deep neural networks are popular for supervised learning applications viz., classification, regression, etc. Besides the type of deep learning architecture, some other decision criteria and parameter selection decisions are required for determining each layer size, number of layers, activation and loss functions for different layers, optimizer algorithm, regularization, etc. Thus, this paper aims to cover different choices available under each of these major and minor decision criteria for devising a neural network and to train it optimally for achieving the objectives effectively, e.g., malware detection, natural language processing, image recognition, etc.

Exemplar models are useful and deep neural networks overcome their limitations: A commentary on Ambridge (2020)

2020 ◽  
Vol 40 (5-6) ◽  
pp. 612-615
Author(s):  
James L. McClelland
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Machine Intelligence ◽  
Human Language ◽  
Language Usage ◽  
Primitive Elements ◽  
Exemplar Models ◽  
Learning Architectures

Humans are sensitive to the properties of individual items, and exemplar models are useful for capturing this sensitivity. I am a proponent of an extension of exemplar-based architectures that I briefly describe. However, exemplar models are very shallow architectures in which it is necessary to stipulate a set of primitive elements that make up each example, and such architectures have not been as successful as deep neural networks in capturing language usage and meaning. More work is needed bringing contemporary deep learning architectures used in machine intelligence to the effort to understand human language processing.

scholarly journals Chatbots Employing Deep Learning for Big Data

2019 ◽  
Vol 8 (11) ◽  
pp. 1005-1010
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Heterogeneous Data ◽  
Instructive Feedback ◽  
Technical Specifications

With the evolution of artificial intelligence to deep learning, the age of perspicacious machines has pioneered that can even mimic as a human. A Conversational software agent is one of the best-suited examples of such intuitive machines which are also commonly known as chatbot actuated with natural language processing. The paper enlisted some existing popular chatbots along with their details, technical specifications, and functionalities. Research shows that most of the customers have experienced penurious service. Also, the inception of meaningful cum instructive feedback endure a demanding and exigent assignment as enactment for chatbots builtout reckon mostly upon templates and hand-written rules. Current chatbot models lack in generating required responses and thus contradict the quality conversation. So involving deep learning amongst these models can overcome this lack and can fill up the paucity with deep neural networks. Some of the deep Neural networks utilized for this till now are Stacked Auto-Encoder, sparse auto-encoders, predictive sparse and denoising auto-encoders. But these DNN are unable to handle big data involving large amounts of heterogeneous data. While Tensor Auto Encoder which overcomes this drawback is time-consuming. This paper has proposed the Chatbot to handle the big data in a manageable time.

scholarly journals Multi-Modal Pain Intensity Assessment Based on Physiological Signals: A Deep Learning Perspective

2021 ◽  
Vol 12 ◽  
Author(s):  
Patrick Thiam ◽  
Heinke Hihn ◽  
Daniel A. Braun ◽  
Hans A. Kestler ◽  
Friedhelm Schwenker
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Pain Intensity ◽  
Supervised Learning ◽  
Pain Assessment ◽  
Deep Neural Networks ◽  
Fine Tuning ◽  
Physiological Data ◽  
Intensity Assessment ◽  
Learning Techniques

Traditional pain assessment approaches ranging from self-reporting methods, to observational scales, rely on the ability of an individual to accurately assess and successfully report observed or experienced pain episodes. Automatic pain assessment tools are therefore more than desirable in cases where this specific ability is negatively affected by various psycho-physiological dispositions, as well as distinct physical traits such as in the case of professional athletes, who usually have a higher pain tolerance as regular individuals. Hence, several approaches have been proposed during the past decades for the implementation of an autonomous and effective pain assessment system. These approaches range from more conventional supervised and semi-supervised learning techniques applied on a set of carefully hand-designed feature representations, to deep neural networks applied on preprocessed signals. Some of the most prominent advantages of deep neural networks are the ability to automatically learn relevant features, as well as the inherent adaptability of trained deep neural networks to related inference tasks. Yet, some significant drawbacks such as requiring large amounts of data to train deep models and over-fitting remain. Both of these problems are especially relevant in pain intensity assessment, where labeled data is scarce and generalization is of utmost importance. In the following work we address these shortcomings by introducing several novel multi-modal deep learning approaches (characterized by specific supervised, as well as self-supervised learning techniques) for the assessment of pain intensity based on measurable bio-physiological data. While the proposed supervised deep learning approach is able to attain state-of-the-art inference performances, our self-supervised approach is able to significantly improve the data efficiency of the proposed architecture by automatically generating physiological data and simultaneously performing a fine-tuning of the architecture, which has been previously trained on a significantly smaller amount of data.

scholarly journals Neural Networks for Entity Matching: A Survey

2021 ◽  
Vol 15 (3) ◽  
pp. 1-37
Author(s):  
Nils Barlaug ◽  
Jon Atle Gulla
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Challenging Problem ◽  
Entity Matching ◽  
New Methods ◽  
Matching Process ◽  
Learning Techniques

Entity matching is the problem of identifying which records refer to the same real-world entity. It has been actively researched for decades, and a variety of different approaches have been developed. Even today, it remains a challenging problem, and there is still generous room for improvement. In recent years, we have seen new methods based upon deep learning techniques for natural language processing emerge. In this survey, we present how neural networks have been used for entity matching. Specifically, we identify which steps of the entity matching process existing work have targeted using neural networks, and provide an overview of the different techniques used at each step. We also discuss contributions from deep learning in entity matching compared to traditional methods, and propose a taxonomy of deep neural networks for entity matching.

scholarly journals Automatic Detection of Arrhythmia Based on Multi-Resolution Representation of ECG Signal

Sensors ◽  
2020 ◽  
Vol 20 (6) ◽  
pp. 1579
Author(s):  
Dongqi Wang ◽  
Qinghua Meng ◽  
Dongming Chen ◽  
Hupo Zhang ◽  
Lisheng Xu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Channel Model ◽  
Expert Knowledge ◽  
Automatic Detection ◽  
Data Representation ◽  
Learning Technology ◽  
Arrhythmia Detection ◽  
Automatic Feature Extraction

Automatic detection of arrhythmia is of great significance for early prevention and diagnosis of cardiovascular disease. Traditional feature engineering methods based on expert knowledge lack multidimensional and multi-view information abstraction and data representation ability, so the traditional research on pattern recognition of arrhythmia detection cannot achieve satisfactory results. Recently, with the increase of deep learning technology, automatic feature extraction of ECG data based on deep neural networks has been widely discussed. In order to utilize the complementary strength between different schemes, in this paper, we propose an arrhythmia detection method based on the multi-resolution representation (MRR) of ECG signals. This method utilizes four different up to date deep neural networks as four channel models for ECG vector representations learning. The deep learning based representations, together with hand-crafted features of ECG, forms the MRR, which is the input of the downstream classification strategy. The experimental results of big ECG dataset multi-label classification confirm that the F1 score of the proposed method is 0.9238, which is 1.31%, 0.62%, 1.18% and 0.6% higher than that of each channel model. From the perspective of architecture, this proposed method is highly scalable and can be employed as an example for arrhythmia recognition.

scholarly journals Enabling deeper learning on big data for materials informatics applications

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Dipendra Jha ◽  
Vishu Gupta ◽  
Logan Ward ◽  
Zijiang Yang ◽  
Christopher Wolverton ◽  
...  
Keyword(s):  
Neural Networks ◽  
Big Data ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Materials Science ◽  
Prediction Models ◽  
Model Performance ◽  
Materials Informatics ◽  
Learning Framework ◽  
Significant Attention

AbstractThe application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

scholarly journals Laplacian networks: bounding indicator function smoothness for neural networks robustness

2021 ◽  
Vol 10 ◽  
Author(s):  
Carlos Lassance ◽  
Vincent Gripon ◽  
Antonio Ortega
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Supervised Learning ◽  
Indicator Function ◽  
Training Data ◽  
Theoretical Justification ◽  
The Past ◽  
Noisy Examples

For the past few years, deep learning (DL) robustness (i.e. the ability to maintain the same decision when inputs are subject to perturbations) has become a question of paramount importance, in particular in settings where misclassification can have dramatic consequences. To address this question, authors have proposed different approaches, such as adding regularizers or training using noisy examples. In this paper we introduce a regularizer based on the Laplacian of similarity graphs obtained from the representation of training data at each layer of the DL architecture. This regularizer penalizes large changes (across consecutive layers in the architecture) in the distance between examples of different classes, and as such enforces smooth variations of the class boundaries. We provide theoretical justification for this regularizer and demonstrate its effectiveness to improve robustness on classical supervised learning vision datasets for various types of perturbations. We also show it can be combined with existing methods to increase overall robustness.

scholarly journals A Survey on Bias in Deep NLP

2021 ◽  
Vol 11 (7) ◽  
pp. 3184
Author(s):  
Ismael Garrido-Muñoz  ◽  
Arturo Montejo-Ráez  ◽  
Fernando Martínez-Santiago  ◽  
L. Alfonso Ureña-López 
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Natural Language Processing ◽  
Probability Distribution ◽  
Natural Language ◽  
Network Design ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Learning Processes ◽  
Relevant Issue

Deep neural networks are hegemonic approaches to many machine learning areas, including natural language processing (NLP). Thanks to the availability of large corpora collections and the capability of deep architectures to shape internal language mechanisms in self-supervised learning processes (also known as “pre-training”), versatile and performing models are released continuously for every new network design. These networks, somehow, learn a probability distribution of words and relations across the training collection used, inheriting the potential flaws, inconsistencies and biases contained in such a collection. As pre-trained models have been found to be very useful approaches to transfer learning, dealing with bias has become a relevant issue in this new scenario. We introduce bias in a formal way and explore how it has been treated in several networks, in terms of detection and correction. In addition, available resources are identified and a strategy to deal with bias in deep NLP is proposed.

Sign in / Sign up

Export Citation Format

Share Document