scholarly journals KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

2021 ◽  
Vol 9 ◽  
pp. 176-194
Author(s):  
Xiaozhi Wang ◽  
Tianyu Gao ◽  
Zhaocheng Zhu ◽  
Zhengyan Zhang ◽  
Zhiyuan Liu ◽  
...  
Keyword(s):  
Link Prediction ◽  
Large Scale ◽  
State Of The Art ◽  
Source Code ◽  
Language Modeling ◽  
Unified Model ◽  
Factual Knowledge ◽  
Textual Information ◽  
Language Representation ◽  
Knowledge Graphs

Abstract Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M1 , a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

scholarly journals TransET: Knowledge Graph Embedding with Entity Types

Electronics ◽  
2021 ◽  
Vol 10 (12) ◽  
pp. 1407
Author(s):  
Peng Wang ◽  
Jing Zhou ◽  
Yuzhang Liu ◽  
Xingchen Zhou
Keyword(s):  
Link Prediction ◽  
State Of The Art ◽  
Score Function ◽  
Graph Embedding ◽  
Vector Spaces ◽  
Knowledge Graph ◽  
Semantic Features ◽  
Knowledge Graphs ◽  
Real World Datasets ◽  
Low Dimensional

Knowledge graph embedding aims to embed entities and relations into low-dimensional vector spaces. Most existing methods only focus on triple facts in knowledge graphs. In addition, models based on translation or distance measurement cannot fully represent complex relations. As well-constructed prior knowledge, entity types can be employed to learn the representations of entities and relations. In this paper, we propose a novel knowledge graph embedding model named TransET, which takes advantage of entity types to learn more semantic features. More specifically, circle convolution based on the embeddings of entity and entity types is utilized to map head entity and tail entity to type-specific representations, then translation-based score function is used to learn the presentation triples. We evaluated our model on real-world datasets with two benchmark tasks of link prediction and triple classification. Experimental results demonstrate that it outperforms state-of-the-art models in most cases.

scholarly journals OpenBioLink: a benchmarking framework for large-scale biomedical link prediction

2020 ◽  
Vol 36 (13) ◽  
pp. 4097-4098 ◽  
Author(s):  
Anna Breit ◽  
Simon Ott ◽  
Asan Agibetov ◽  
Matthias Samwald
Keyword(s):  
Link Prediction ◽  
Large Scale ◽  
Source Code ◽  
Machine Learning Algorithms ◽  
Knowledge Networks ◽  
Supplementary Information ◽  
Supplementary Data ◽  
Biomedical Knowledge ◽  
High Quality ◽  
Baseline Evaluation

Abstract Summary Recently, novel machine-learning algorithms have shown potential for predicting undiscovered links in biomedical knowledge networks. However, dedicated benchmarks for measuring algorithmic progress have not yet emerged. With OpenBioLink, we introduce a large-scale, high-quality and highly challenging biomedical link prediction benchmark to transparently and reproducibly evaluate such algorithms. Furthermore, we present preliminary baseline evaluation results. Availability and implementation Source code and data are openly available at https://github.com/OpenBioLink/OpenBioLink. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Link Prediction via Ranking Metric Dual-Level Attention Network Learning

2017 ◽  
Author(s):  
Zhou Zhao ◽  
Ben Gao ◽  
Vincent W. Zheng ◽  
Deng Cai ◽  
Xiaofei He ◽  
...  
Keyword(s):  
Link Prediction ◽  
Large Scale ◽  
State Of The Art ◽  
Feature Representation ◽  
Telecommunication Networks ◽  
Metric Embedding ◽  
Network Learning ◽  
Learning Framework ◽  
Discriminative Feature ◽  
Attentional Learning

Link prediction is a challenging problem for complex network analysis, arising in many disciplines such as social networks and telecommunication networks. Currently, many existing approaches estimate the proximity of the link endpoints for link prediction from their feature or the local neighborhood around them, which suffer from the localized view of network connections and insufficiency of discriminative feature representation. In this paper, we consider the problem of link prediction from the viewpoint of learning discriminative path-based proximity ranking metric embedding. We propose a novel ranking metric network learning framework by jointly exploiting both node-level and path-level attentional proximity of the endpoints for link prediction. We then develop the path-based dual-level reasoning attentional learning method with recurrent neural network for proximity ranking metric embedding. The extensive experiments on two large-scale datasets show that our method achieves better performance than other state-of-the-art solutions to the problem.

scholarly journals DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning

2017 ◽  
Author(s):  
Xiaodong Gu ◽  
Hongyu Zhang ◽  
Dongmei Zhang ◽  
Sunghun Kim
Keyword(s):  
Sequence Learning ◽  
Large Scale ◽  
Intelligent System ◽  
State Of The Art ◽  
Source Code ◽  
Computer Programs ◽  
Experimental Results ◽  
Multiple Devices ◽  
Application Programming ◽  
Programming Interfaces

Computer programs written in one language are often required to be ported to other languages to support multiple devices and environments. When programs use language specific APIs (Application Programming Interfaces), it is very challenging to migrate these APIs to the corresponding APIs written in other languages. Existing approaches mine API mappings from projects that have corresponding versions in two languages. They rely on the sparse availability of bilingual projects, thus producing a limited number of API mappings. In this paper, we propose an intelligent system called DeepAM for automatically mining API mappings from a large-scale code corpus without bilingual projects. The key component of DeepAM is based on the multi-modal sequence to sequence learning architecture that aims to learn joint semantic representations of bilingual API sequences from big source code data. Experimental results indicate that DeepAM significantly increases the accuracy of API mappings as well as the number of API mappings when compared with the state-of-the-art approaches.

scholarly journals Guided Generation of Cause and Effect

2020 ◽  
Author(s):  
Zhongyang Li ◽  
Xiao Ding ◽  
Ting Liu ◽  
J. Edward Hu ◽  
Benjamin Van Durme
Keyword(s):  
Causal Reasoning ◽  
Large Scale ◽  
State Of The Art ◽  
Prior Work ◽  
Text Generation ◽  
Causal Knowledge ◽  
High Quality ◽  
Cause And Effect ◽  
Knowledge Graphs ◽  
Human Assessment

We present a conditional text generation framework that posits sentential expressions of possible causes and effects. This framework depends on two novel resources we develop in the course of this work: a very large-scale collection of English sentences expressing causal patterns (CausalBank); and a refinement over previous work on constructing large lexical causal knowledge graphs (Cause Effect Graph). Further, we extend prior work in lexically-constrained decoding to support disjunctive positive constraints. Human assessment confirms that our approach gives high-quality and diverse outputs. Finally, we use CausalBank to perform continued training of an encoder supporting a recent state-of-the-art model for causal reasoning, leading to a 3-point improvement on the COPA challenge set, with no change in model architecture.

scholarly journals Summarizing Source Code with Transferred API Knowledge

2018 ◽  
Author(s):  
Xing Hu ◽  
Ge Li ◽  
Xin Xia ◽  
David Lo ◽  
Shuai Lu ◽  
...  
Keyword(s):  
Real World ◽  
Software Maintenance ◽  
Large Scale ◽  
State Of The Art ◽  
Source Code ◽  
Code Search ◽  
Novel Approach ◽  
Software Maintenance And Evolution ◽  
World Industry ◽  
Similar Code

Code summarization, aiming to generate succinct natural language description of source code, is extremely useful for code search and code comprehension. It has played an important role in software maintenance and evolution. Previous approaches generate summaries by retrieving summaries from similar code snippets. However, these approaches heavily rely on whether similar code snippets can be retrieved, how similar the snippets are, and fail to capture the API knowledge in the source code, which carries vital information about the functionality of the source code. In this paper, we propose a novel approach, named TL-CodeSum, which successfully uses API knowledge learned in a different but related task to code summarization. Experiments on large-scale real-world industry Java projects indicate that our approach is effective and outperforms the state-of-the-art in code summarization.

scholarly journals To hop or not, that is the question: Towards effective multi-hop reasoning over knowledge graphs

2021 ◽  
Author(s):  
Jinzhi Liao ◽  
Xiang Zhao ◽  
Jiuyang Tang ◽  
Weixin Zeng ◽  
Zhen Tan
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Supervised Learning ◽  
Large Scale ◽  
State Of The Art ◽  
False Negative ◽  
Stop Signal ◽  
Knowledge Graph ◽  
Overall Performance ◽  
Knowledge Graphs

AbstractWith the proliferation of large-scale knowledge graphs (KGs), multi-hop knowledge graph reasoning has been a capstone that enables machines to be able to handle intelligent tasks, especially where some explicit reasoning path is appreciated for decision making. To train a KG reasoner, supervised learning-based methods suffer from false-negative issues, i.e., unseen paths during training are not to be found in prediction; in contrast, reinforcement learning (RL)-based methods do not require labeled paths, and can explore to cover many appropriate reasoning paths. In this connection, efforts have been dedicated to investigating several RL formulations for multi-hop KG reasoning. Particularly, current RL-based methods generate rewards at the very end of the reasoning process, due to which short paths of hops less than a given threshold are likely to be overlooked, and the overall performance is impaired. To address the problem, we propose , a revised RL formulation of multi-hop KG reasoning that is characterized by two novel designs—the stop signal and the worth-trying signal. The stop signal instructs the agent of RL to stay at the entity after finding the answer, preventing from hopping further even if the threshold is not reached; meanwhile, the worth-trying signal encourages the agent to try to learn some partial patterns from the paths that fail to lead to the answer. To validate the design of our model , comprehensive experiments are carried out on three benchmark knowledge graphs, and the results and analysis suggest the superiority of over state-of-the-art methods.

scholarly journals Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training

2020 ◽  
Vol 34 (07) ◽  
pp. 11336-11344 ◽  
Author(s):  
Gen Li ◽  
Nan Duan ◽  
Yuejian Fang ◽  
Ming Gong ◽  
Daxin Jiang
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Language Modeling ◽  
Context Aware ◽  
Commonsense Reasoning ◽  
Output Layer ◽  
The Cross ◽  
Cross Lingual ◽  
Vision And Language ◽  
Image Caption

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM (Lample and Conneau 2019) and Unicoder (Huang et al. 2019), both visual and linguistic contents are fed into a multi-layer Transformer (Vaswani et al. 2017) for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling(MLM), Masked Object Classification(MOC) and Visual-linguistic Matching(VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.

scholarly journals SNEQ: Semi-Supervised Attributed Network Embedding with Attention-Based Quantisation

2020 ◽  
Vol 34 (04) ◽  
pp. 4091-4098 ◽  
Author(s):  
Tao He ◽  
Lianli Gao ◽  
Jingkuan Song ◽  
Xin Wang ◽  
Kejie Huang ◽  
...  
Keyword(s):  
Link Prediction ◽  
Large Scale ◽  
State Of The Art ◽  
Network Embedding ◽  
Compression Method ◽  
Network Analytics ◽  
Attributed Network ◽  
Large Scale Networks ◽  
Low Dimensional ◽  
Embedding Methods

Learning accurate low-dimensional embeddings for a network is a crucial task as it facilitates many network analytics tasks. Moreover, the trained embeddings often require a significant amount of space to store, making storage and processing a challenge, especially as large-scale networks become more prevalent. In this paper, we present a novel semi-supervised network embedding and compression method, SNEQ, that is competitive with state-of-art embedding methods while being far more space- and time-efficient. SNEQ incorporates a novel quantisation method based on a self-attention layer that is trained in an end-to-end fashion, which is able to dramatically compress the size of the trained embeddings, thus reduces storage footprint and accelerates retrieval speed. Our evaluation on four real-world networks of diverse characteristics shows that SNEQ outperforms a number of state-of-the-art embedding methods in link prediction, node classification and node recommendation. Moreover, the quantised embedding shows a great advantage in terms of storage and time compared with continuous embeddings as well as hashing methods.

Large Scale Graph Mining with MapReduce

2011 ◽  
pp. 66-78 ◽  
Author(s):  
Charalampos E. Tsourakakis
Keyword(s):  
Survey Research ◽  
Present State ◽  
Graph Mining ◽  
Large Scale ◽  
State Of The Art ◽  
Source Code ◽  
Research Work

In this chapter, the authors present state of the art work on large scale graph mining using MapReduce. They survey research work on an important graph mining problem, estimating the diameter of a graph and the eccentricities/radii of its vertices. Thanks to the algorithm they present in the following, the authors are able to mine graphs with billions of edges, and thus extract surprising patterns. The source code is publicly available at the URL http://www.cs.cmu.edu/~pegasus/.

Sign in / Sign up

Export Citation Format

Share Document