A Survey on Cognitive Packet Networks: Taxonomy, State-of-the-art, Recurrent Neural Networks, and QoS Metrics

<p>SMILES randomization, a form of data augmentation, has previously been shown to increase the performance of deep learning models compared to non-augmented baselines. Here, we propose a novel data augmentation method we call “Levenshtein augmentation” which considers local SMILES sub-sequence similarity between reactants and their respective products when creating training pairs. The performance of Levenshtein augmentation was tested using two state of the art models - transformer and sequence-to-sequence based recurrent neural networks with attention. Levenshtein augmentation demonstrated an increase performance over non-augmented, and conventionally SMILES randomization augmented data when used for training of baseline models. Furthermore, Levenshtein augmentation seemingly results in what we define as <i>attentional gain </i>– an enhancement in the pattern recognition capabilities of the underlying network to molecular motifs.</p>

Download Full-text

Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

10.1101/2021.03.22.436372 ◽

2021 ◽

Author(s):

Bojian Yin ◽

Federico Corradi ◽

Sander M. Bohté

Keyword(s):

Neural Networks ◽

Time Domain ◽

Recurrent Neural Networks ◽

State Of The Art ◽

Spiking Neurons ◽

Recurrent Networks ◽

Computationally Efficient ◽

Hardware Implementations ◽

Comparable Performance ◽

The Time Domain

ABSTRACTInspired by more detailed modeling of biological neurons, Spiking neural networks (SNNs) have been investigated both as more biologically plausible and potentially more powerful models of neural computation, and also with the aim of extracting biological neurons’ energy efficiency; the performance of such networks however has remained lacking compared to classical artificial neural networks (ANNs). Here, we demonstrate how a novel surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields state-of-the-art for SNNs on challenging benchmarks in the time-domain, like speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks (RNNs) and approaches that of the best modern ANNs. As these SNNs exhibit sparse spiking, we show that they theoretically are one to three orders of magnitude more computationally efficient compared to RNNs with comparable performance. Together, this positions SNNs as an attractive solution for AI hardware implementations.

Download Full-text

An empirical study on temporal modeling for online action detection

Complex & Intelligent Systems ◽

10.1007/s40747-021-00534-3 ◽

2021 ◽

Author(s):

Wen Wang ◽

Xiaojiang Peng ◽

Yu Qiao ◽

Jian Cheng

Keyword(s):

Neural Networks ◽

Empirical Study ◽

Recurrent Neural Networks ◽

State Of The Art ◽

Deep Convolutional Neural Networks ◽

Temporal Modeling ◽

Action Detection ◽

Modeling Methods ◽

Feature Extractor ◽

First Time

AbstractOnline action detection (OAD) is a practical yet challenging task, which has attracted increasing attention in recent years. A typical OAD system mainly consists of three modules: a frame-level feature extractor which is usually based on pre-trained deep Convolutional Neural Networks (CNNs), a temporal modeling module, and an action classifier. Among them, the temporal modeling module is crucial which aggregates discriminative information from historical and current features. Though many temporal modeling methods have been developed for OAD and other topics, their effects are lack of investigation on OAD fairly. This paper aims to provide an empirical study on temporal modeling for OAD including four meta types of temporal modeling methods, i.e. temporal pooling, temporal convolution, recurrent neural networks, and temporal attention, and uncover some good practices to produce a state-of-the-art OAD system. Many of them are explored in OAD for the first time, and extensively evaluated with various hyper parameters. Furthermore, based on our empirical study, we present several hybrid temporal modeling methods. Our best networks, i.e. , the hybridization of DCC, LSTM and M-NL, and the hybridization of DCC and M-NL, which outperform previously published results with sizable margins on THUMOS-14 dataset (48.6% vs. 47.2%) and TVSeries dataset (84.3% vs. 83.7%).

Download Full-text

Skeleton-Based Action Recognition with Joint Coordinates as Feature Using Neural Oblivious Decision Ensembles

10.3233/faia210037 ◽

2021 ◽

Author(s):

Fakhrul Aniq Hakimi Nasrul ’Alam ◽

Mohd. Ibrahim Shapiai ◽

Uzma Batool ◽

Ahmad Kamal Ramli ◽

Khairil Ashraf Elias

Keyword(s):

Neural Networks ◽

Action Recognition ◽

Recurrent Neural Networks ◽

State Of The Art ◽

Structured Data ◽

Joint Coordinates ◽

Label Prediction ◽

Within Subjects ◽

High Degree ◽

Behaviour Recognition

Recognition of human behavior is critical in video monitoring, human-computer interaction, video comprehension, and virtual reality. The key problem with behaviour recognition in video surveillance is the high degree of variation between and within subjects. Numerous studies have suggested background-insensitive skeleton-based as the proven detection technique. The present state-of-the-art approaches to skeleton-based action recognition rely primarily on Recurrent Neural Networks (RNN) and Convolution Neural Networks (CNN). Both methods take dynamic human skeleton as the input to the network. We chose to handle skeleton data differently, relying solely on its skeleton joint coordinates as the input. The skeleton joints’ positions are defined in (x, y) coordinates. In this paper, we investigated the incorporation of the Neural Oblivious Decision Ensemble (NODE) into our proposed action classifier network. The skeleton is extracted using a pose estimation technique based on the Residual Network (ResNet). It extracts the 2D skeleton of 18 joints for each detected body. The joint coordinates of the skeleton are stored in a table in the form of rows and columns. Each row represents the position of the joints. The structured data are fed into NODE for label prediction. With the proposed network, we obtain 97.5% accuracy on RealWorld (HAR) dataset. Experimental results show that the proposed network outperforms one the state-of-the-art approaches by 1.3%. In conclusion, NODE is a promising deep learning technique for structured data analysis as compared to its machine learning counterparts such as the GBDT packages; Catboost, and XGBoost.

Download Full-text

DialogueRNN: An Attentive RNN for Emotion Detection in Conversations

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33016818 ◽

2019 ◽

Vol 33 ◽

pp. 6818-6825 ◽

Cited By ~ 27

Author(s):

Navonil Majumder ◽

Soujanya Poria ◽

Devamanyu Hazarika ◽

Rada Mihalcea ◽

Alexander Gelbukh ◽

...

Keyword(s):

Neural Networks ◽

Social Media ◽

Recurrent Neural Networks ◽

Opinion Mining ◽

State Of The Art ◽

Emotion Detection ◽

Emotion Classification ◽

Argumentation Mining ◽

Consumer Feedback ◽

The Individual

Emotion detection in conversations is a necessary step for a number of applications, including opinion mining over chat history, social media threads, debates, argumentation mining, understanding consumer feedback in live conversations, and so on. Currently systems do not treat the parties in the conversation individually by adapting to the speaker of each utterance. In this paper, we describe a new method based on recurrent neural networks that keeps track of the individual party states throughout the conversation and uses this information for emotion classification. Our model outperforms the state-of-the-art by a significant margin on two different datasets.

Download Full-text

Sentence embeddings in NLI with iterative refinement encoders

Natural Language Engineering ◽

10.1017/s1351324919000202 ◽

2019 ◽

Vol 25 (4) ◽

pp. 467-482 ◽

Cited By ~ 3

Author(s):

Aarne Talman ◽

Anssi Yli-Jyrä ◽

Jörg Tiedemann

Keyword(s):

Neural Networks ◽

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

Recurrent Neural Networks ◽

State Of The Art ◽

Iterative Refinement ◽

Learning Tasks ◽

Sentence Level ◽

Refinement Strategy

AbstractSentence-level representations are necessary for various natural language processing tasks. Recurrent neural networks have proven to be very effective in learning distributed representations and can be trained efficiently on natural language inference tasks. We build on top of one such model and propose a hierarchy of bidirectional LSTM and max pooling layers that implements an iterative refinement strategy and yields state of the art results on the SciTail dataset as well as strong results for Stanford Natural Language Inference and Multi-Genre Natural Language Inference. We can show that the sentence embeddings learned in this way can be utilized in a wide variety of transfer learning tasks, outperforming InferSent on 7 out of 10 and SkipThought on 8 out of 9 SentEval sentence embedding evaluation tasks. Furthermore, our model beats the InferSent model in 8 out of 10 recently published SentEval probing tasks designed to evaluate sentence embeddings’ ability to capture some of the important linguistic properties of sentences.

Download Full-text

Stateful Premise Selection by Recurrent Neural Networks

10.29007/j5hd ◽

2020 ◽

Author(s):

Bartosz Piotrowski ◽

Josef Urban

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Data Augmentation ◽

State Of The Art ◽

Language Translation ◽

New Method ◽

New Learning ◽

Augmentation Techniques

In this work we develop a new learning-based method for selecting facts (premises) when proving new goals over large formal libraries. Unlike previous methods that choose sets of facts independently of each other by their rank, the new method uses the notion of state that is updated each time a choice of a fact is made. Our stateful architecture is based on recurrent neural networks which have been recently very successful in stateful tasks such as language translation. The new method is combined with data augmentation techniques, evaluated in several ways on a standard large-theory benchmark and compared to state-of-the-art premise approach based on gradient boosted trees. It is shown to perform significantly better and to solve many new problems.

Download Full-text

Classifying relations in clinical narratives using segment graph convolutional and recurrent neural networks (Seg-GCRNs)

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocy157 ◽

2018 ◽

Vol 26 (3) ◽

pp. 262-268 ◽

Cited By ~ 13

Author(s):

Yifu Li ◽

Ran Jin ◽

Yuan Luo

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

State Of The Art ◽

Medical Problem ◽

Word Embedding ◽

Feature Engineering ◽

Medical Test ◽

Relation Classification ◽

Syntactic Dependencies ◽

Syntactic Dependency

Abstract We propose to use segment graph convolutional and recurrent neural networks (Seg-GCRNs), which use only word embedding and sentence syntactic dependencies, to classify relations from clinical notes without manual feature engineering. In this study, the relations between 2 medical concepts are classified by simultaneously learning representations of text segments in the context of sentence syntactic dependency: preceding, concept1, middle, concept2, and succeeding segments. Seg-GCRN was systematically evaluated on the i2b2/VA relation classification challenge datasets. Experiments show that Seg-GCRN attains state-of-the-art micro-averaged F-measure for all 3 relation categories: 0.692 for classifying medical treatment–problem relations, 0.827 for medical test–problem relations, and 0.741 for medical problem–medical problem relations. Comparison with the previous state-of-the-art segment convolutional neural network (Seg-CNN) suggests that adding syntactic dependency information helps refine medical word embedding and improves concept relation classification without manual feature engineering. Seg-GCRN can be trained efficiently for the i2b2/VA dataset on a GPU platform.

Download Full-text

DGA CapsNet: 1D Application of Capsule Networks to DGA Detection

Information ◽

10.3390/info10050157 ◽

2019 ◽

Vol 10 (5) ◽

pp. 157 ◽

Cited By ~ 5

Author(s):

Daniel S. Berman

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Real Time ◽

Convolutional Neural Networks ◽

Recurrent Neural Networks ◽

State Of The Art ◽

One Dimensional ◽

Domain Names ◽

Large Numbers ◽

And Control

Domain generation algorithms (DGAs) represent a class of malware used to generate large numbers of new domain names to achieve command-and-control (C2) communication between the malware program and its C2 server to avoid detection by cybersecurity measures. Deep learning has proven successful in serving as a mechanism to implement real-time DGA detection, specifically through the use of recurrent neural networks (RNNs) and convolutional neural networks (CNNs). This paper compares several state-of-the-art deep-learning implementations of DGA detection found in the literature with two novel models: a deeper CNN model and a one-dimensional (1D) Capsule Networks (CapsNet) model. The comparison shows that the 1D CapsNet model performs as well as the best-performing model from the literature.

Download Full-text

Text classification by untrained sentence embeddings

Intelligenza Artificiale ◽

10.3233/ia-200053 ◽

2021 ◽

Vol 14 (2) ◽

pp. 245-259

Author(s):

Daniele Di Sarli ◽

Claudio Gallicchio ◽

Alessio Micheli

Keyword(s):

Neural Networks ◽

Text Classification ◽

Recurrent Neural Networks ◽

Learning Algorithm ◽

State Of The Art ◽

Sequential Data ◽

Training Time ◽

Reference Models ◽

Gating Mechanisms ◽

Heavy Training

Recurrent Neural Networks (RNNs) represent a natural paradigm for modeling sequential data like text written in natural language. In fact, RNNs and their variations have long been the architecture of choice in many applications, however in practice they require the use of labored architectures (such as gating mechanisms) and computationally heavy training processes. In this paper we address the question of whether it is possible to generate sentence embeddings via completely untrained recurrent dynamics, on top of which to apply a simple learning algorithm for text classification. This would allow to obtain extremely efficient models in terms of training time. Our work investigates the extent to which this approach can be used, by analyzing the results on different tasks. Finally, we show that, within certain limits, it is possible to build extremely efficient models for text classification that remain competitive in accuracy with reference models in the state-of-the-art.

Download Full-text