Jointly Modeling Heterogeneous Student Behaviors and Interactions among Multiple Prediction Tasks

Haobing Liu; Yanmin Zhu; Tianzi Zang; Yanan Xu; Jiadi Yu; Feilong Tang

doi:10.1145/3458023

Jointly Modeling Heterogeneous Student Behaviors and Interactions among Multiple Prediction Tasks

ACM Transactions on Knowledge Discovery from Data ◽

10.1145/3458023 ◽

2021 ◽

Vol 16 (1) ◽

pp. 1-24

Author(s):

Haobing Liu ◽

Yanmin Zhu ◽

Tianzi Zang ◽

Yanan Xu ◽

Jiadi Yu ◽

...

Keyword(s):

Student Affairs ◽

Short Term Memory ◽

Attention Mechanism ◽

Practical Significance ◽

Behavior Prediction ◽

Student Behaviors ◽

Student Profile ◽

Heterogeneous Behavior ◽

Multiple Prediction ◽

Digital Footprints

Prediction tasks about students have practical significance for both student and college. Making multiple predictions about students is an important part of a smart campus. For instance, predicting whether a student will fail to graduate can alert the student affairs office to take predictive measures to help the student improve his/her academic performance. With the development of information technology in colleges, we can collect digital footprints that encode heterogeneous behaviors continuously. In this article, we focus on modeling heterogeneous behaviors and making multiple predictions together, since some prediction tasks are related and learning the model for a specific task may have the data sparsity problem. To this end, we propose a variant of Long-Short Term Memory (LSTM) and a soft-attention mechanism. The proposed LSTM is able to learn the student profile-aware representation from heterogeneous behavior sequences. The proposed soft-attention mechanism can dynamically learn different importance degrees of different days for every student. In this way, heterogeneous behaviors can be well modeled. In order to model interactions among multiple prediction tasks, we propose a co-attention mechanism based unit. With the help of the stacked units, we can explicitly control the knowledge transfer among multiple tasks. We design three motivating behavior prediction tasks based on a real-world dataset collected from a college. Qualitative and quantitative experiments on the three prediction tasks have demonstrated the effectiveness of our model.

End-to-End Chinese Dialect Discrimination with Self-Attention

10.5121/csit.2021.111425 ◽

2021 ◽

Author(s):

Yangjie Dan ◽

Fan Xu ◽

Mingwen Wang

Keyword(s):

Large Scale ◽

Short Term Memory ◽

Attention Mechanism ◽

Practical Significance ◽

Sequence Information ◽

Discrimination Model ◽

Chinese Dialect ◽

End To End ◽

Low Performance ◽

Phonetic Features

Dialect discrimination has an important practical significance for protecting inheritance of dialects. The traditional dialect discrimination methods pay much attention to the underlying acoustic features, and ignore the meaning of the pronunciation itself, resulting in low performance. This paper systematically explores the validity of the pronunciation features of dialect speech composed of phoneme sequence information for dialect discrimination, and designs an end-to-end dialect discrimination model based on the multi-head self-attention mechanism. Specifically, we first adopt the residual convolution neural network and the multihead self-attention mechanism to effectively extract the phoneme sequence features unique to different dialects to compose the novel phonetic features. Then, we perform dialect discrimination based on the extracted phonetic features using the self-attention mechanism and bi-directional long short-term memory networks. The experimental results on the large-scale benchmark 10-way Chinese dialect corpus released by IFLYTEK 1 show that our model outperforms the state-of-the-art alternatives by large margin.

Dam Deformation Interpretation and Prediction Based on a Long Short-Term Memory Model Coupled with an Attention Mechanism

Applied Sciences ◽

10.3390/app11146625 ◽

2021 ◽

Vol 11 (14) ◽

pp. 6625

Author(s):

Yan Su ◽

Kailiang Weng ◽

Chuan Lin ◽

Zeqin Chen

Keyword(s):

Short Term Memory ◽

Attention Mechanism ◽

Impact Factors ◽

Nonlinear Prediction ◽

Time Dimension ◽

Short Term ◽

Deformation Prediction ◽

Term Memory ◽

Long Short Term Memory ◽

Dam Deformation

An accurate dam deformation prediction model is vital to a dam safety monitoring system, as it helps assess and manage dam risks. Most traditional dam deformation prediction algorithms ignore the interpretation and evaluation of variables and lack qualitative measures. This paper proposes a data processing framework that uses a long short-term memory (LSTM) model coupled with an attention mechanism to predict the deformation response of a dam structure. First, the random forest (RF) model is introduced to assess the relative importance of impact factors and screen input variables. Secondly, the density-based spatial clustering of applications with noise (DBSCAN) method is used to identify and filter the equipment based abnormal values to reduce the random error in the measurements. Finally, the coupled model is used to focus on important factors in the time dimension in order to obtain more accurate nonlinear prediction results. The results of the case study show that, of all tested methods, the proposed coupled method performed best. In addition, it was found that temperature and water level both have significant impacts on dam deformation and can serve as reliable metrics for dam management.

Abnormal Detection of Electricity Consumption of User Based on Particle Swarm Optimization and Long Short Term Memory With the Attention Mechanism

IEEE Access ◽

10.1109/access.2021.3062675 ◽

2021 ◽

Vol 9 ◽

pp. 47252-47265

Author(s):

Jiahao Bian ◽

Lei Wang ◽

Rafal Scherer ◽

Marcin Wozniak ◽

Pengchao Zhang ◽

...

Keyword(s):

Particle Swarm Optimization ◽

Short Term Memory ◽

Particle Swarm ◽

Electricity Consumption ◽

Attention Mechanism ◽

Short Term ◽

Swarm Optimization ◽

Term Memory ◽

Abnormal Detection ◽

Long Short Term Memory

A Method Based on Attention Mechanism using Bidirectional Long-Short Term Memory(BLSTM) for Question Answering

10.1109/icee52715.2021.9544258 ◽

2021 ◽

Author(s):

Seyed Vahid Moravvej ◽

Mohammad Javad Maleki Kahaki ◽

Moein Salimi Sartakhti ◽

Abdolreza Mirzaei

Keyword(s):

Question Answering ◽

Short Term Memory ◽

Attention Mechanism ◽

Short Term ◽

Term Memory ◽

Long Short Term Memory

LAR: A User Behavior Prediction Model in Server Log Based on LSTM-Attention Network and RSC Algorithm

Fuzzy Systems and Data Mining VI - Frontiers in Artificial Intelligence and Applications ◽

10.3233/faia200709 ◽

2020 ◽

Author(s):

Yingying Shang

Keyword(s):

Clustering Algorithm ◽

Short Term Memory ◽

User Behavior ◽

Absolute Error ◽

Research Area ◽

Behavior Prediction ◽

Attention Network ◽

Important Research Area ◽

User Access ◽

Single User

Using server log data to predict the URLs that a user is likely to visit is an important research area in user behavior prediction. In this paper, a predictive model (called LAR) based on the long short-term memory (LSTM) attention network and reciprocal-nearest-neighbors supported clustering algorithm (RSC) for predicting the URL is proposed. First, the LSTM-attention network is used to predict the URL categories a user might visit, and the RSC algorithm is then used to cluster users. Subsequently, the URLs belonging to the same category are determined from the user clusters to predict the URLs that the user might visit. The proposed LAR model considers the time sequence of the user access URL, and the relationship between a single user and group users, which effectively improves the prediction accuracy. The experimental results demonstrate that the LAR model is feasible and effective for user behavior prediction. The accuracy of the mean absolute error and root mean square error of the LAR model are better than those of the other models compared in this study.

Predicting Outcomes of Business Process Executions Based on LSTM Neural Networks and Attention Mechanism

10.21203/rs.3.rs-260970/v1 ◽

2021 ◽

Author(s):

Jiaojiao Wang ◽

Dongjin Yu ◽

Chengfei Liu ◽

Xiaoxiao Sun

Keyword(s):

Short Term Memory ◽

Attention Mechanism ◽

Sequential Data ◽

Time Prediction ◽

Prediction Time ◽

Highly Sensitive ◽

Long Short Term Memory ◽

Early Decision ◽

Lstm Network ◽

Hidden Layer

Abstract To effectively predict the outcome of an on-going process instance helps make an early decision, which plays an important role in so-called predictive process monitoring. Existing methods in this field are tailor-made for some empirical operations such as the prefix extraction, clustering, and encoding, leading that their relative accuracy is highly sensitive to the dataset. Moreover, they have limitations in real-time prediction applications due to the lengthy prediction time. Since Long Short-term Memory (LSTM) neural network provides a high precision in the prediction of sequential data in several areas, this paper investigates LSTM and its enhancements and proposes three different approaches to build more effective and efficient models for outcome prediction. The first move on enhancement is that we combine the original LSTM network from two directions, forward and backward, to capture more features from the completed cases. The second move on enhancement is that we add attention mechanism after extracting features in the hidden layer of LSTM network to distinct them from their attention weight. A series of extensive experiments are evaluated on twelve real datasets when comparing with other approaches. The results show that our approaches outperform the state-of-the-art ones in terms of prediction effectiveness and time performance.

Combined Self-Attention Mechanism for Chinese Named Entity Recognition in Military

Future Internet ◽

10.3390/fi11080180 ◽

2019 ◽

Vol 11 (8) ◽

pp. 180

Author(s):

Fei Liao ◽

Liangli Ma ◽

Jingjing Pei ◽

Linshan Tan

Keyword(s):

Domain Knowledge ◽

Short Term Memory ◽

Contextual Information ◽

Named Entity Recognition ◽

Attention Mechanism ◽

The Self ◽

Entity Recognition ◽

Named Entity ◽

Vector Representations ◽

The Military

Military named entity recognition (MNER) is one of the key technologies in military information extraction. Traditional methods for the MNER task rely on cumbersome feature engineering and specialized domain knowledge. In order to solve this problem, we propose a method employing a bidirectional long short-term memory (BiLSTM) neural network with a self-attention mechanism to identify the military entities automatically. We obtain distributed vector representations of the military corpus by unsupervised learning and the BiLSTM model combined with the self-attention mechanism is adopted to capture contextual information fully carried by the character vector sequence. The experimental results show that the self-attention mechanism can improve effectively the performance of MNER task. The F-score of the military documents and network military texts identification was 90.15% and 89.34%, respectively, which was better than other models.

CWPC_BiAtt: Character–Word–Position Combined BiLSTM-Attention for Chinese Named Entity Recognition

Information ◽

10.3390/info11010045 ◽

2020 ◽

Vol 11 (1) ◽

pp. 45 ◽

Cited By ~ 1

Author(s):

Shardrom Johnson ◽

Sherlock Shen ◽

Yuanchen Liu

Keyword(s):

Language Processing ◽

Short Term Memory ◽

Conditional Random Field ◽

Named Entity Recognition ◽

Attention Mechanism ◽

Entity Recognition ◽

Position Information ◽

Named Entity ◽

Pos Tagging ◽

Word Position

Usually taken as linguistic features by Part-Of-Speech (POS) tagging, Named Entity Recognition (NER) is a major task in Natural Language Processing (NLP). In this paper, we put forward a new comprehensive-embedding, considering three aspects, namely character-embedding, word-embedding, and pos-embedding stitched in the order we give, and thus get their dependencies, based on which we propose a new Character–Word–Position Combined BiLSTM-Attention (CWPC_BiAtt) for the Chinese NER task. Comprehensive-embedding via the Bidirectional Llong Short-Term Memory (BiLSTM) layer can get the connection between the historical and future information, and then employ the attention mechanism to capture the connection between the content of the sentence at the current position and that at any location. Finally, we utilize Conditional Random Field (CRF) to decode the entire tagging sequence. Experiments show that CWPC_BiAtt model we proposed is well qualified for the NER task on Microsoft Research Asia (MSRA) dataset and Weibo NER corpus. A high precision and recall were obtained, which verified the stability of the model. Position-embedding in comprehensive-embedding can compensate for attention-mechanism to provide position information for the disordered sequence, which shows that comprehensive-embedding has completeness. Looking at the entire model, our proposed CWPC_BiAtt has three distinct characteristics: completeness, simplicity, and stability. Our proposed CWPC_BiAtt model achieved the highest F-score, achieving the state-of-the-art performance in the MSRA dataset and Weibo NER corpus.

Daily runoff forecasting based on data-augmented neural network model

Journal of Hydroinformatics ◽

10.2166/hydro.2020.017 ◽

2020 ◽

Vol 22 (4) ◽

pp. 900-915 ◽

Cited By ~ 2

Author(s):

Xiao-ying Bi ◽

Bo Li ◽

Wen-long Lu ◽

Xin-zhi Zhou

Keyword(s):

Neural Network ◽

Data Augmentation ◽

Short Term Memory ◽

Interpolation Method ◽

Linear Interpolation ◽

Attention Mechanism ◽

Ar Model ◽

Support Vector ◽

Runoff Forecasting ◽

Comparison Results

Abstract Accurate daily runoff prediction plays an important role in the management and utilization of water resources. In order to improve the accuracy of prediction, this paper proposes a deep neural network (CAGANet) composed of a convolutional layer, an attention mechanism, a gated recurrent unit (GRU) neural network, and an autoregressive (AR) model. Given that the daily runoff sequence is abrupt and unstable, it is difficult for a single model and combined model to obtain high-precision daily runoff predictions directly. Therefore, this paper uses a linear interpolation method to enhance the stability of hydrological data and apply the augmented data to the CAGANet model, the support vector machine (SVM) model, the long short-term memory (LSTM) neural network and the attention-mechanism-based LSTM model (AM-LSTM). The comparison results show that among the four models based on data augmentation, the CAGANet model proposed in this paper has the best prediction accuracy. Its Nash–Sutcliffe efficiency can reach 0.993. Therefore, the CAGANet model based on data augmentation is a feasible daily runoff forecasting scheme.

Emotion-Semantic-Enhanced Bidirectional LSTM with Multi-Head Attention Mechanism for Microblog Sentiment Analysis

Information ◽

10.3390/info11050280 ◽

2020 ◽

Vol 11 (5) ◽

pp. 280

Author(s):

Shaoxiu Wang ◽

Yonghua Zhu ◽

Wenjing Gao ◽

Meng Cao ◽

Mengyao Li

Keyword(s):

Sentiment Analysis ◽

Short Term Memory ◽

Syntactic Structure ◽

Contextual Information ◽

Research Field ◽

Attention Mechanism ◽

Feature Representation ◽

Mechanism Model ◽

Hidden Layer ◽

The Impact

The sentiment analysis of microblog text has always been a challenging research field due to the limited and complex contextual information. However, most of the existing sentiment analysis methods for microblogs focus on classifying the polarity of emotional keywords while ignoring the transition or progressive impact of words in different positions in the Chinese syntactic structure on global sentiment, as well as the utilization of emojis. To this end, we propose the emotion-semantic-enhanced bidirectional long short-term memory (BiLSTM) network with the multi-head attention mechanism model (EBILSTM-MH) for sentiment analysis. This model uses BiLSTM to learn feature representation of input texts, given the word embedding. Subsequently, the attention mechanism is used to assign the attentive weights of each words to the sentiment analysis based on the impact of emojis. The attentive weights can be combined with the output of the hidden layer to obtain the feature representation of posts. Finally, the sentiment polarity of microblog can be obtained through the dense connection layer. The experimental results show the feasibility of our proposed model on microblog sentiment analysis when compared with other baseline models.