Modeling low- and high-order feature interactions with FM and self-attention network

2020 ◽  
Author(s):  
Cairong Yan ◽  
Yizhou Chen ◽  
Yongquan Wan ◽  
Pengwei Wang
Keyword(s):  
High Order ◽  
Attention Network ◽  
Feature Interactions

scholarly journals An Attention-Based Latent Information Extraction Network (ALIEN) for High-Order Feature Interactions

2020 ◽  
Vol 10 (16) ◽  
pp. 5468
Author(s):  
Ruo Huang ◽  
Shelby McIntyre ◽  
Meina Song ◽  
Haihong E ◽  
Zhonghong Ou
Keyword(s):  
Information Extraction ◽  
Recommender Systems ◽  
Vital Role ◽  
High Order ◽  
Sequential Patterns ◽  
Sequence Information ◽  
Recommendation Algorithm ◽  
Feature Interactions ◽  
Real World Datasets ◽  
Click Through Rate

One of the primary tasks for commercial recommender systems is to predict the probabilities of users clicking items, e.g., advertisements, music and products. This is because such predictions have a decisive impact on profitability. The classic recommendation algorithm, collaborative filtering (CF), still plays a vital role in many industrial recommender systems. However, although straight CF is good at capturing similar users’ preferences for items based on their past interactions, it lacks regarding (1) modeling the influences of users’ sequential patterns from their individual history interaction sequences and (2) the relevance of users’ and items’ attributes. In this work, we developed an attention-based latent information extraction network (ALIEN) for click-through rate prediction, to integrate (1) implicit user similarity in terms of click patterns (analogous to CF), and (2) modeling the low and high-order feature interactions and (3) historical sequence information. The new model is based on the deep learning, which goes beyond the capabilities of econometric approaches, such as matrix factorization (MF) and k-means. In addition, the approach provides explainability to the recommendation by interpreting the contributions of different features and historical interactions. We have conducted experiments on real-world datasets that demonstrate considerable improvements over strong baselines.

scholarly journals PMDFI: Predicting miRNA–Disease Associations Based on High-Order Feature Interaction

2021 ◽  
Vol 12 ◽  
Author(s):  
Mingyan Tang ◽  
Chenzhe Liu ◽  
Dayun Liu ◽  
Junyi Liu ◽  
Jiaqi Liu ◽  
...  
Keyword(s):  
Computational Models ◽  
Treatment Method ◽  
High Order ◽  
Computational Method ◽  
Feature Interaction ◽  
Rna Molecules ◽  
Feature Interactions ◽  
Non Coding Rna ◽  
Disease Associations ◽  
Fold Cross Validation

MicroRNAs (miRNAs) are non-coding RNA molecules that make a significant contribution to diverse biological processes, and their mutations and dysregulations are closely related to the occurrence, development, and treatment of human diseases. Therefore, identification of potential miRNA–disease associations contributes to elucidating the pathogenesis of tumorigenesis and seeking the effective treatment method for diseases. Due to the expensive cost of traditional biological experiments of determining associations between miRNAs and diseases, increasing numbers of effective computational models are being used to compensate for this limitation. In this study, we propose a novel computational method, named PMDFI, which is an ensemble learning method to predict potential miRNA–disease associations based on high-order feature interactions. We initially use a stacked autoencoder to extract meaningful high-order features from the original similarity matrix, and then perform feature interactive learning, and finally utilize an integrated model composed of multiple random forests and logistic regression to make comprehensive predictions. The experimental results illustrate that PMDFI achieves excellent performance in predicting potential miRNA–disease associations, with the average area under the ROC curve scores of 0.9404 and 0.9415 in 5-fold and 10-fold cross-validation, respectively.

scholarly journals LCHI: Low-Order Correlation and High-Order Interaction Integrated Model Oriented to Network Intrusion Detection

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Shengwei Lei ◽  
Chunhe Xia ◽  
Tianbo Wang
Keyword(s):  
Intrusion Detection ◽  
High Order ◽  
Network Intrusion Detection ◽  
Feature Interaction ◽  
Detection Accuracy ◽  
High Order Interaction ◽  
Order Interaction ◽  
Feature Interactions ◽  
Network Intrusion ◽  
Low Order

Network intrusion poses a severe threat to the Internet of Things (IoT). Thus, it is essential to study information security protection technology in IoT. Learning sophisticated feature interactions is critical in improving detection accuracy for network intrusion. Despite significant progress, existing methods seem to have a strong bias towards single low- or high-order feature interaction. Moreover, they always extract all possible low-order interactions indiscriminately, introducing too much noise. To address the above problems, we propose a low-order correlation and high-order interaction (LCHI) integrated feature extraction model. First, we selectively extract the beneficial low-order correlation between the same-type features by the multivariate correlation analysis (MCA) model and attention mechanism. Second, we extract the complicated high-order feature interaction by the deep neural network (DNN) model. Finally, we emphasize both the low- and high-order feature interactions and incorporate them. Our LCHI model seamlessly combines the linearity of MCA in modeling lower-order feature correlation and the nonlinearity of DNN in modeling higher-order feature interaction. Conceptually, our LCHI is more expressive than the previous models. We carry on a series of experiments on the public wireless and wired network intrusion detection datasets. The experimental results show that LCHI improves 1.06%, 2.46%, 3.74%, 0.25%, 1.17%, and 0.64% on the AWID, NSL-KDD, UNSW-NB15, CICIDS 2017, CICIDS 2018, and DAPT 2020 datasets, respectively.

scholarly journals End-to-End Bootstrapping Neural Network for Entity Set Expansion

2020 ◽  
Vol 34 (05) ◽  
pp. 9402-9409
Author(s):  
Lingyong Yan ◽  
Xianpei Han ◽  
Ben He ◽  
Le Sun
Keyword(s):  
Neural Network ◽  
Recurrent Neural Network ◽  
Substantial Improvement ◽  
High Order ◽  
Experimental Results ◽  
Target Category ◽  
Attention Network ◽  
Order Relations ◽  
Decoder Architecture ◽  
End To End

Bootstrapping for entity set expansion (ESE) has long been modeled as a multi-step pipelined process. Such a paradigm, unfortunately, often suffers from two main challenges: 1) the entities are expanded in multiple separate steps, which tends to introduce noisy entities and results in the semantic drift problem; 2) it is hard to exploit the high-order entity-pattern relations for entity set expansion. In this paper, we propose an end-to-end bootstrapping neural network for entity set expansion, named BootstrapNet, which models the bootstrapping in an encoder-decoder architecture. In the encoding stage, a graph attention network is used to capture both the first- and the high-order relations between entities and patterns, and encode useful information into their representations. In the decoding stage, the entities are sequentially expanded through a recurrent neural network, which outputs entities at each stage, and its hidden state vectors, representing the target category, are updated at each expansion step. Experimental results demonstrate substantial improvement of our model over previous ESE approaches.

Deep context interaction network based on attention mechanism for click-through rate prediction

2021 ◽  
pp. 1-16
Author(s):  
Ling Yuan ◽  
Zhuwen Pan ◽  
Ping Sun ◽  
Yinzhen Wei ◽  
Haiping Yu
Keyword(s):  
Online Advertising ◽  
Dimensional Space ◽  
Interaction Network ◽  
High Order ◽  
Attention Mechanism ◽  
Feature Interaction ◽  
Feature Interactions ◽  
Low Dimensional ◽  
Click Through Rate ◽  
The Relationship

Click-through rate (CTR) prediction, which aims to predict the probability of a user clicking on an ad, is a critical task in online advertising systems. The problem is very challenging since(1) an effective prediction relies on high-order combinatorial features, and(2)the relationship to auxiliary ads that may impact the CTR. In this paper, we propose Deep Context Interaction Network on Attention Mechanism(DCIN-Attention) to process feature interaction and context at the same time. The context includes other ads in the current search page, historically clicked and unclicked ads of the user. Specifically, we use the attention mechanism to learn the interactions between the target ad and each type of auxiliary ad. The residual network is used to model the feature interactions in the low-dimensional space, and with the multi-head self-attention neural network, high-order feature interactions can be modeled. Experimental results on Avito dataset show that DCIN outperform several existing methods for CTR prediction.

Sign in / Sign up

Export Citation Format

Share Document