scholarly journals Attention-based hierarchical recurrent neural networks for MOOC forum posts analysis

2020 ◽  
Author(s):  
Nicola Capuano ◽  
Santi Caballé ◽  
Jordi Conesa ◽  
Antonio Greco
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Text Categorization ◽  
Online Courses ◽  
Recurrent Network ◽  
Subject Area ◽  
Conversational Agents ◽  
Collaborative Tools ◽  
Experimental Findings ◽  
Productive Talk

AbstractMassive open online courses (MOOCs) allow students and instructors to discuss through messages posted on a forum. However, the instructors should limit their interaction to the most critical tasks during MOOC delivery so, teacher-led scaffolding activities, such as forum-based support, can be very limited, even impossible in such environments. In addition, students who try to clarify the concepts through such collaborative tools could not receive useful answers, and the lack of interactivity may cause a permanent abandonment of the course. The purpose of this paper is to report the experimental findings obtained evaluating the performance of a text categorization tool capable of detecting the intent, the subject area, the domain topics, the sentiment polarity, and the level of confusion and urgency of a forum post, so that the result may be exploited by instructors to carefully plan their interventions. The proposed approach is based on the application of attention-based hierarchical recurrent neural networks, in which both a recurrent network for word encoding and an attention mechanism for word aggregation at sentence and document levels are used before classification. The integration of the developed classifier inside an existing tool for conversational agents, based on the academically productive talk framework, is also presented as well as the accuracy of the proposed method in the classification of forum posts.

Recurrent Neural Networks with Small Weights Implement Definite Memory Machines

2003 ◽  
Vol 15 (8) ◽  
pp. 1897-1929 ◽  
Author(s):  
Barbara Hammer ◽  
Peter Tiňo
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Experimental Studies ◽  
Activation Function ◽  
Transition Function ◽  
Recurrent Network ◽  
Point Of View ◽  
Recurrent Networks ◽  
Contraction Parameter ◽  
Arbitrary Precision

Recent experimental studies indicate that recurrent neural networks initialized with “small” weights are inherently biased toward definite memory machines (Tiňno, Čerňanský, & Beňušková, 2002a, 2002b). This article establishes a theoretical counterpart: transition function of recurrent network with small weights and squashing activation function is a contraction. We prove that recurrent networks with contractive transition function can be approximated arbitrarily well on input sequences of unbounded length by a definite memory machine. Conversely, every definite memory machine can be simulated by a recurrent network with contractive transition function. Hence, initialization with small weights induces an architectural bias into learning with recurrent neural networks. This bias might have benefits from the point of view of statistical learning theory: it emphasizes one possible region of the weight space where generalization ability can be formally proved. It is well known that standard recurrent neural networks are not distribution independent learnable in the probably approximately correct (PAC) sense if arbitrary precision and inputs are considered. We prove that recurrent networks with contractive transition function with a fixed contraction parameter fulfill the so-called distribution independent uniform convergence of empirical distances property and hence, unlike general recurrent networks, are distribution independent PAC learnable.

scholarly journals An Optimal Feature Parameter Set Based on Gated Recurrent Unit Recurrent Neural Networks for Speech Segment Detection

2020 ◽  
Vol 10 (4) ◽  
pp. 1273 ◽  
Author(s):  
Özlem BATUR DİNLER ◽  
Nizamettin AYDIN
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Feature Vector ◽  
Processing Parameters ◽  
Speech Segmentation ◽  
Hybrid Features ◽  
Speech Segment ◽  
Experimental Findings ◽  
Gated Recurrent Unit ◽  
Optimal Feature

Speech segment detection based on gated recurrent unit (GRU) recurrent neural networks for the Kurdish language was investigated in the present study. The novelties of the current research are the utilization of a GRU in Kurdish speech segment detection, creation of a unique database from the Kurdish language, and optimization of processing parameters for Kurdish speech segmentation. This study is the first attempt to find the optimal feature parameters of the model and to form a large Kurdish vocabulary dataset for a speech segment detection based on consonant, vowel, and silence (C/V/S) discrimination. For this purpose, four window sizes and three window types with three hybrid feature vector techniques were used to describe the phoneme boundaries. Identification of the phoneme boundaries using a GRU recurrent neural network was performed with six different classification algorithms for the C/V/S discrimination. We have demonstrated that the GRU model has achieved outstanding speech segmentation performance for characterizing Kurdish acoustic signals. The experimental findings of the present study show the significance of the segment detection of speech signals by effectively utilizing hybrid features, window sizes, window types, and classification models for Kurdish speech.

scholarly journals Representation of Linguistic Form and Function in Recurrent Neural Networks

2017 ◽  
Vol 43 (4) ◽  
pp. 761-780 ◽  
Author(s):  
Ákos Kádár ◽  
Grzegorz Chrupała ◽  
Afra Alishahi
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Network Architecture ◽  
Language Model ◽  
Visual Pathway ◽  
Recurrent Network ◽  
Point Of View ◽  
Language Models ◽  
Grammatical Function ◽  
Form And Function

We present novel methods for analyzing the activation patterns of recurrent neural networks from a linguistic point of view and explore the types of linguistic structure they learn. As a case study, we use a standard standalone language model, and a multi-task gated recurrent network architecture consisting of two parallel pathways with shared word embeddings: The Visual pathway is trained on predicting the representations of the visual scene corresponding to an input sentence, and the Textual pathway is trained to predict the next word in the same sentence. We propose a method for estimating the amount of contribution of individual tokens in the input to the final prediction of the networks. Using this method, we show that the Visual pathway pays selective attention to lexical categories and grammatical functions that carry semantic information, and learns to treat word types differently depending on their grammatical function and their position in the sequential structure of the sentence. In contrast, the language models are comparatively more sensitive to words with a syntactic function. Further analysis of the most informative n-gram contexts for each model shows that in comparison with the Visual pathway, the language models react more strongly to abstract contexts that represent syntactic constructions.

scholarly journals The Prediction Process Based on Deep Recurrent Neural Networks: A Review

2021 ◽  
pp. 29-45
Author(s):  
Diyar Qader Zeebaree ◽  
Adnan Mohsin Abdulazeez ◽  
Lozan M. Abdullrhman ◽  
Dathar Abas Hasan ◽  
Omar Sedqi Kareem
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Recurrent Network ◽  
Current Layer ◽  
Recurrent Networks ◽  
Daily Lives ◽  
Long Short Term Memory ◽  
The Impact ◽  
Very High

Prediction is vital in our daily lives, as it is used in various ways, such as learning, adapting, predicting, and classifying. The prediction of parameters capacity of RNNs is very high; it provides more accurate results than the conventional statistical methods for prediction. The impact of a hierarchy of recurrent neural networks on Predicting process is studied in this paper. A recurrent network takes the hidden state of the previous layer as input and generates as output the hidden state of the current layer. Some of deep Learning algorithms can be utilized in as prediction tools in video analysis, musical information retrieval and time series applications. Recurrent networks may process examples simultaneously, maintaining a state or memory that recreates an arbitrarily long background window. Long Short-Term Memory (LSTM) and Bidirectional RNN (BRNN) are examples of recurrent networks. This paper aims to give a comprehensive assessment of predictions based on RNN. Additionally, each paper presents all relevant facts, such as dataset, method, architecture, and the accuracy of the predictions they deliver.

scholarly journals A MANY-TO-MANY FULLY CONVOLUTIONAL RECURRENT NETWORK FOR MULTITEMPORAL CROP RECOGNITION

2019 ◽  
Vol IV-2/W7 ◽  
pp. 25-32 ◽  
Author(s):  
J. A. Chamorro ◽  
J. D. Bermudez ◽  
P. N. Happ ◽  
R. Q. Feitosa
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Temporal Dynamics ◽  
Agricultural Practices ◽  
Remote Sensing Data ◽  
Recurrent Network ◽  
Data Sets ◽  
Tropical Regions ◽  
Spatio Temporal ◽  
Multitemporal Remote Sensing

<p><strong>Abstract.</strong> Recently, recurrent neural networks have been proposed for crop mapping from multitemporal remote sensing data. Most of these proposals have been designed and tested in temperate regions, where a single harvest per season is the rule. In tropical regions, the favorable climate and local agricultural practices, such as crop rotation, result in more complex spatio-temporal dynamics, where the single harvest per season assumption does not hold. In this context, a demand arises for methods capable of recognizing agricultural crops at multiple dates along the multitemporal sequence. In the present work, we propose to adapt two recurrent neural networks, originally conceived for single harvest per season, for multidate crop recognition. In addition, we propose a novel multidate approach based on bidirectional fully convolutional recurrent neural networks. These three architectures were evaluated on public Sentinel-1 data sets from two tropical regions in Brazil. In our experiments, all methods achieved state-of-the-art accuracies with a clear superiority of the proposed architecture. It outperformed its counterparts in up to 3.8% and 7.4%, in terms of per-month overall accuracy, and it was the best performing method in terms of F1-score for most crops and dates on both regions.</p>

scholarly journals Music emotion recognition using recurrent neural networks and pretrained models

2021 ◽  
Author(s):  
Jacek Grekow
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Recurrent Network ◽  
Sequential Data ◽  
Circumplex Model ◽  
Learning Networks ◽  
Audio Features ◽  
Svm Algorithm ◽  
Audio Feature Extraction

AbstractThe article presents conducted experiments using recurrent neural networks for emotion detection in musical segments. Trained regression models were used to predict the continuous values of emotions on the axes of Russell’s circumplex model. A process of audio feature extraction and creating sequential data for learning networks with long short-term memory (LSTM) units is presented. Models were implemented using the WekaDeeplearning4j package and a number of experiments were carried out with data with different sets of features and varying segmentation. The usefulness of dividing the data into sequences as well as the point of using recurrent networks to recognize emotions in music, the results of which have even exceeded the SVM algorithm for regression, were demonstrated. The author analyzed the effect of the network structure and the set of used features on the results of the regressors recognizing values on two axes of the emotion model: arousal and valence. Finally, the use of a pretrained model for processing audio features and training a recurrent network with new sequences of features is presented.

Sign in / Sign up

Export Citation Format

Share Document