A deep embedding model for knowledge graph completion based on attention mechanism

2021 ◽  
Author(s):  
Jin Huang ◽  
TingHua Zhang ◽  
Jia Zhu ◽  
Weihao Yu ◽  
Yong Tang ◽  
...  
Keyword(s):  
Attention Mechanism ◽  
Knowledge Graph ◽  
Deep Embedding

scholarly journals Relational Graph Neural Network with Hierarchical Attention for Knowledge Graph Completion

2020 ◽  
Vol 34 (05) ◽  
pp. 9612-9619
Author(s):  
Zhao Zhang ◽  
Fuzhen Zhuang ◽  
Hengshu Zhu ◽  
Zhiping Shi ◽  
Hui Xiong ◽  
...  
Keyword(s):  
Neural Network ◽  
Missing Values ◽  
State Of The Art ◽  
Attention Mechanism ◽  
Knowledge Graph ◽  
Incomplete Knowledge ◽  
Neighborhood Information ◽  
Local Neighborhood ◽  
Proposed Model ◽  
Knowledge Graphs

The rapid proliferation of knowledge graphs (KGs) has changed the paradigm for various AI-related applications. Despite their large sizes, modern KGs are far from complete and comprehensive. This has motivated the research in knowledge graph completion (KGC), which aims to infer missing values in incomplete knowledge triples. However, most existing KGC models treat the triples in KGs independently without leveraging the inherent and valuable information from the local neighborhood surrounding an entity. To this end, we propose a Relational Graph neural network with Hierarchical ATtention (RGHAT) for the KGC task. The proposed model is equipped with a two-level attention mechanism: (i) the first level is the relation-level attention, which is inspired by the intuition that different relations have different weights for indicating an entity; (ii) the second level is the entity-level attention, which enables our model to highlight the importance of different neighboring entities under the same relation. The hierarchical attention mechanism makes our model more effective to utilize the neighborhood information of an entity. Finally, we extensively validate the superiority of RGHAT against various state-of-the-art baselines.

scholarly journals Knowledge Graph Alignment Network with Gated Multi-Hop Neighborhood Aggregation

2020 ◽  
Vol 34 (01) ◽  
pp. 222-229
Author(s):  
Zequn Sun ◽  
Chengming Wang ◽  
Wei Hu ◽  
Muhao Chen ◽  
Jian Dai ◽  
...  
Keyword(s):  
Neural Networks ◽  
Attention Mechanism ◽  
Knowledge Graph ◽  
Neighborhood Information ◽  
Graph Alignment ◽  
Gating Mechanism ◽  
Neighborhood Structures ◽  
Knowledge Graphs ◽  
End To End ◽  
Graph Neural Networks

Graph neural networks (GNNs) have emerged as a powerful paradigm for embedding-based entity alignment due to their capability of identifying isomorphic subgraphs. However, in real knowledge graphs (KGs), the counterpart entities usually have non-isomorphic neighborhood structures, which easily causes GNNs to yield different representations for them. To tackle this problem, we propose a new KG alignment network, namely AliNet, aiming at mitigating the non-isomorphism of neighborhood structures in an end-to-end manner. As the direct neighbors of counterpart entities are usually dissimilar due to the schema heterogeneity, AliNet introduces distant neighbors to expand the overlap between their neighborhood structures. It employs an attention mechanism to highlight helpful distant neighbors and reduce noises. Then, it controls the aggregation of both direct and distant neighborhood information using a gating mechanism. We further propose a relation loss to refine entity representations. We perform thorough experiments with detailed ablation studies and analyses on five entity alignment datasets, demonstrating the effectiveness of AliNet.

HSRec: Hierarchical self-attention incorporating knowledge graph for sequential recommendation

2021 ◽  
pp. 1-12
Author(s):  
Zuoxi Yang ◽  
Shoubin Dong
Keyword(s):  
Semantic Information ◽  
High Order ◽  
Attention Mechanism ◽  
Sequential Patterns ◽  
Knowledge Graph ◽  
Intrinsic Interest ◽  
Fine Grained ◽  
Potential Interest ◽  
Preference Evolution ◽  
Real World Datasets

Modeling user’s fine-grained preferences and dynamic preference evolution from their chronological behaviors are challenging and crucial for sequential recommendation. In this paper, we develop a Hierarchical Self-Attention Incorporating Knowledge Graph for Sequential Recommendation (HSRec). HSRec models not only the user’s intrinsic preferences but also the user’s external potential interests to capture the user’s fine-grained preferences. Specifically, the intrinsic interest module and potential interest module are designed to capture these two preferences respectively. In the intrinsic interest module, user’s sequential patterns are characterized from their behaviors via the self-attention mechanism. As for the potential interest module, high-order paths can be generated with the help of the knowledge graph. Therefore, a hierarchical self-attention mechanism is designed to aggregate the semantic information of user interaction from these paths. Specifically, an entity-level self-attention mechanism is applied to capture the sequential patterns contained in the high-order paths while an interaction-level self-attention mechanism is designed to further capture the semantic information from user interactions. Moreover, according to the high-order semantic relevance, HSRec can explore the user’s dynamic preferences at each time, thus describing the user’s dynamic preference evolution. Finally, experiments conducted on three real world datasets demonstrate the state-of-the-art performance of the HSRec.

scholarly journals Attention-Based Graph Convolutional Network for Zero-Shot Learning with Pre-Training

2021 ◽  
Vol 2021 ◽  
pp. 1-13
Author(s):  
Xuefei Wu ◽  
Mingjiang Liu ◽  
Bo Xin ◽  
Zhangqing Zhu ◽  
Gang Wang
Keyword(s):  
Loss Function ◽  
Attention Mechanism ◽  
Propagation Model ◽  
Knowledge Graph ◽  
Learning Paradigm ◽  
Convolutional Network ◽  
Dense Graph ◽  
Convolutional Networks ◽  
Central Node ◽  
Graph Propagation

Zero-shot learning (ZSL) is a powerful and promising learning paradigm for classifying instances that have not been seen in training. Although graph convolutional networks (GCNs) have recently shown great potential for the ZSL tasks, these models cannot adjust the constant connection weights between the nodes in knowledge graph and the neighbor nodes contribute equally to classify the central node. In this study, we apply an attention mechanism to adjust the connection weights adaptively to learn more important information for classifying unseen target nodes. First, we propose an attention graph convolutional network for zero-shot learning (AGCNZ) by integrating the attention mechanism and GCN directly. Then, in order to prevent the dilution of knowledge from distant nodes, we apply the dense graph propagation (DGP) model for the ZSL tasks and propose an attention dense graph propagation model for zero-shot learning (ADGPZ). Finally, we propose a modified loss function with a relaxation factor to further improve the performance of the learned classifier. Experimental results under different pre-training settings verified the effectiveness of the proposed attention-based models for ZSL.

scholarly journals KGGCN: Knowledge-Guided Graph Convolutional Networks for Distantly Supervised Relation Extraction

2021 ◽  
Vol 11 (16) ◽  
pp. 7734
Author(s):  
Ningyi Mao ◽  
Wenti Huang ◽  
Hai Zhong
Keyword(s):  
Prior Knowledge ◽  
Structural Information ◽  
Relation Extraction ◽  
Attention Mechanism ◽  
Knowledge Graph ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Lexical Resource ◽  
Sentence Level ◽  
The Impact

Distantly supervised relation extraction is the most popular technique for identifying semantic relation between two entities. Most prior models only focus on the supervision information present in training sentences. In addition to training sentences, external lexical resource and knowledge graphs often contain other relevant prior knowledge. However, relation extraction models usually ignore such readily available information. Moreover, previous works only utilize a selective attention mechanism over sentences to alleviate the impact of noise, they lack the consideration of the implicit interaction between sentences with relation facts. In this paper, (1) a knowledge-guided graph convolutional network is proposed based on the word-level attention mechanism to encode the sentences. It can capture the key words and cue phrases to generate expressive sentence-level features by attending to the relation indicators obtained from the external lexical resource. (2) A knowledge-guided sentence selector is proposed, which explores the semantic and structural information of triples from knowledge graph as sentence-level knowledge attention to distinguish the importance of each individual sentence. Experimental results on two widely used datasets, NYT-FB and GDS, show that our approach is able to efficiently use the prior knowledge from the external lexical resource and knowledge graph to enhance the performance of distantly supervised relation extraction.

Path-based reasoning approach for knowledge graph completion using CNN-BiLSTM with attention mechanism

2020 ◽  
Vol 142 ◽  
pp. 112960 ◽  
Author(s):  
Batselem Jagvaral ◽  
Wan-Kon Lee ◽  
Jae-Seung Roh ◽  
Min-Sung Kim ◽  
Young-Tack Park
Keyword(s):  
Attention Mechanism ◽  
Knowledge Graph

KRAN: Knowledge Refining Attention Network for Recommendation

2022 ◽  
Vol 16 (2) ◽  
pp. 1-20
Author(s):  
Zhenyu Zhang ◽  
Lei Zhang ◽  
Dingqi Yang ◽  
Liu Yang
Keyword(s):  
State Of The Art ◽  
Negative Impact ◽  
Cold Start ◽  
Attention Mechanism ◽  
Knowledge Graph ◽  
Convolutional Network ◽  
Auxiliary Data ◽  
Attention Network ◽  
Additional Information ◽  
Data Source

Recommender algorithms combining knowledge graph and graph convolutional network are becoming more and more popular recently. Specifically, attributes describing the items to be recommended are often used as additional information. These attributes along with items are highly interconnected, intrinsically forming a Knowledge Graph (KG). These algorithms use KGs as an auxiliary data source to alleviate the negative impact of data sparsity. However, these graph convolutional network based algorithms do not distinguish the importance of different neighbors of entities in the KG, and according to Pareto’s principle, the important neighbors only account for a small proportion. These traditional algorithms can not fully mine the useful information in the KG. To fully release the power of KGs for building recommender systems, we propose in this article KRAN, a Knowledge Refining Attention Network, which can subtly capture the characteristics of the KG and thus boost recommendation performance. We first introduce a traditional attention mechanism into the KG processing, making the knowledge extraction more targeted, and then propose a refining mechanism to improve the traditional attention mechanism to extract the knowledge in the KG more effectively. More precisely, KRAN is designed to use our proposed knowledge-refining attention mechanism to aggregate and obtain the representations of the entities (both attributes and items) in the KG. Our knowledge-refining attention mechanism first measures the relevance between an entity and it’s neighbors in the KG by attention coefficients, and then further refines the attention coefficients using a “richer-get-richer” principle, in order to focus on highly relevant neighbors while eliminating less relevant neighbors for noise reduction. In addition, for the item cold start problem, we propose KRAN-CD, a variant of KRAN, which further incorporates pre-trained KG embeddings to handle cold start items. Experiments show that KRAN and KRAN-CD consistently outperform state-of-the-art baselines across different settings.

scholarly journals Translating Embeddings for Knowledge Graph Completion with Relation Attention Mechanism

2018 ◽  
Author(s):  
Wei Qian ◽  
Cong Fu ◽  
Yu Zhu ◽  
Deng Cai ◽  
Xiaofei He
Keyword(s):  
State Of The Art ◽  
Graph Embedding ◽  
Attention Mechanism ◽  
Human Cognition ◽  
Knowledge Graph ◽  
People First ◽  
Public Datasets ◽  
The Right ◽  
The Given ◽  
Essential Problem

Knowledge graph embedding is an essential problem in knowledge extraction. Recently, translation based embedding models (e.g., TransE) have received increasingly attentions. These methods try to interpret the relations among entities as translations from head entity to tail entity and achieve promising performance on knowledge graph completion. Previous researchers attempt to transform the entity embedding concerning the given relation for distinguishability. Also, they naturally think the relation-related transforming should reflect attention mechanism, which means it should focus on only a part of the attributes. However, we found previous methods are failed with creating attention mechanism, and the reason is that they ignore the hierarchical routine of human cognition. When predicting whether a relation holds between two entities, people first check the category of entities, then they focus on fined-grained relation-related attributes to make the decision. In other words, the attention should take effect on entities filtered by the right category. In this paper, we propose a novel knowledge graph embedding method named TransAt to learn the translation based embedding, relation-related categories of entities and relation-related attention simultaneously. Extensive experiments show that our approach outperforms state-of-the-art methods significantly on public datasets, and our method can learn the true attention varying among relations.

Sign in / Sign up

Export Citation Format

Share Document