scholarly journals Probing Natural Language Inference Models through Semantic Fragments

2020 ◽  
Vol 34 (05) ◽  
pp. 8713-8721
Author(s):  
Kyle Richardson ◽  
Hai Hu ◽  
Lawrence Moss ◽  
Ashish Sabharwal
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Fine Tuning ◽  
Language Understanding ◽  
Inference Models ◽  
Linguistic Understanding ◽  
Benchmark Datasets ◽  
Linguistic Behavior ◽  
Linguistic Models

Do state-of-the-art models for language understanding already have, or can they easily learn, abilities such as boolean coordination, quantification, conditionals, comparatives, and monotonicity reasoning (i.e., reasoning about word substitutions in sentential contexts)? While such phenomena are involved in natural language inference (NLI) and go beyond basic linguistic understanding, it is unclear the extent to which they are captured in existing NLI benchmarks and effectively learned by models. To investigate this, we propose the use of semantic fragments—systematically generated datasets that each target a different semantic phenomenon—for probing, and efficiently improving, such capabilities of linguistic models. This approach to creating challenge datasets allows direct control over the semantic diversity and complexity of the targeted linguistic phenomena, and results in a more precise characterization of a model's linguistic behavior. Our experiments, using a library of 8 such semantic fragments, reveal two remarkable findings: (a) State-of-the-art models, including BERT, that are pre-trained on existing NLI benchmark datasets perform poorly on these new fragments, even though the phenomena probed here are central to the NLI task; (b) On the other hand, with only a few minutes of additional fine-tuning—with a carefully selected learning rate and a novel variation of “inoculation”—a BERT-based model can master all of these logic and monotonicity fragments while retaining its performance on established NLI benchmarks.

scholarly journals Deep Cascade Multi-Task Learning for Slot Filling in Online Shopping Assistant

2019 ◽  
Vol 33 ◽  
pp. 6465-6472 ◽  
Author(s):  
Yu Gong ◽  
Xusheng Luo ◽  
Yu Zhu ◽  
Wenwu Ou ◽  
Zhao Li ◽  
...  
Keyword(s):  
Natural Language ◽  
Knowledge Base ◽  
Online Shopping ◽  
State Of The Art ◽  
Language Understanding ◽  
Dialog Systems ◽  
Named Entity ◽  
Online Test ◽  
Benchmark Datasets ◽  
Slot Filling

Slot filling is a critical task in natural language understanding (NLU) for dialog systems. State-of-the-art approaches treat it as a sequence labeling problem and adopt such models as BiLSTM-CRF. While these models work relatively well on standard benchmark datasets, they face challenges in the context of E-commerce where the slot labels are more informative and carry richer expressions. In this work, inspired by the unique structure of E-commerce knowledge base, we propose a novel multi-task model with cascade and residual connections, which jointly learns segment tagging, named entity tagging and slot filling. Experiments show the effectiveness of the proposed cascade and residual structures. Our model has a 14.6% advantage in F1 score over the strong baseline methods on a new Chinese E-commerce shopping assistant dataset, while achieving competitive accuracies on a standard dataset. Furthermore, online test deployed on such dominant E-commerce platform shows 130% improvement on accuracy of understanding user utterances. Our model has already gone into production in the E-commerce platform.

scholarly journals Large-scale Semantic Parsing without Question-Answer Pairs

2014 ◽  
Vol 2 ◽  
pp. 377-392 ◽  
Author(s):  
Siva Reddy ◽  
Mirella Lapata ◽  
Mark Steedman
Keyword(s):  
Natural Language ◽  
Large Scale ◽  
Graph Matching ◽  
State Of The Art ◽  
The State ◽  
Semantic Parsing ◽  
Matching Problem ◽  
Weak Supervision ◽  
Benchmark Datasets

In this paper we introduce a novel semantic parsing approach to query Freebase in natural language without requiring manual annotations or question-answer pairs. Our key insight is to represent natural language via semantic graphs whose topology shares many commonalities with Freebase. Given this representation, we conceptualize semantic parsing as a graph matching problem. Our model converts sentences to semantic graphs using CCG and subsequently grounds them to Freebase guided by denotations as a form of weak supervision. Evaluation experiments on a subset of the Free917 and WebQuestions benchmark datasets show our semantic parser improves over the state of the art.

SANTM: Efficient Self-attention-driven Network for Text Matching

2022 ◽  
Vol 22 (3) ◽  
pp. 1-21
Author(s):  
Prayag Tiwari ◽  
Amit Kumar Jaiswal ◽  
Sahil Garg ◽  
Ilsun You
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
The State ◽  
Attention Mechanism ◽  
Matching Problems ◽  
Attention Model ◽  
Extra Information ◽  
Textual Entailment ◽  
Benchmark Datasets ◽  
Text Matching

Self-attention mechanisms have recently been embraced for a broad range of text-matching applications. Self-attention model takes only one sentence as an input with no extra information, i.e., one can utilize the final hidden state or pooling. However, text-matching problems can be interpreted either in symmetrical or asymmetrical scopes. For instance, paraphrase detection is an asymmetrical task, while textual entailment classification and question-answer matching are considered asymmetrical tasks. In this article, we leverage attractive properties of self-attention mechanism and proposes an attention-based network that incorporates three key components for inter-sequence attention: global pointwise features, preceding attentive features, and contextual features while updating the rest of the components. Our model follows evaluation on two benchmark datasets cover tasks of textual entailment and question-answer matching. The proposed efficient Self-attention-driven Network for Text Matching outperforms the state of the art on the Stanford Natural Language Inference and WikiQA datasets with much fewer parameters.

scholarly journals Key2Vec: Automatic Ranked Keyphrase Extraction from Scientific Articles using Phrase Embeddings

2018 ◽  
Author(s):  
Debanjan Mahata ◽  
John Kuriakose ◽  
Rajiv Ratn Shah ◽  
Roger Zimmermann
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Keyphrase Extraction ◽  
Text Documents ◽  
Benchmark Datasets

Keyphrase extraction is a fundamental task in natural language processing that facilitates mapping of documents to a set of representative phrases. In this paper, we present an unsupervised technique (Key2Vec) that leverages phrase embeddings for ranking keyphrases extracted from scientific articles. Specifically, we propose an effective way of processing text documents for training multi-word phrase embeddings that are used for thematic representation of scientific articles and ranking of keyphrases extracted from them using theme-weighted PageRank. Evaluations are performed on benchmark datasets producing state-of-the-art results.

scholarly journals Bilateral Multi-Perspective Matching for Natural Language Sentences

2017 ◽  
Author(s):  
Zhiguo Wang ◽  
Wael Hamza ◽  
Radu Florian
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
The State ◽  
Experimental Results ◽  
The Other ◽  
Multiple Perspectives ◽  
Time Step ◽  
Benchmark Datasets ◽  
Sentence Matching ◽  
Fully Connected

Natural language sentence matching is a fundamental technology for a variety of tasks. Previous approaches either match sentences from a single direction or only apply single granular (word-by-word or sentence-by-sentence) matching. In this work, we propose a bilateral multi-perspective matching (BiMPM) model. Given two sentences P and Q, our model first encodes them with a BiLSTM encoder. Next, we match the two encoded sentences in two directions P against Q and P against Q. In each matching direction, each time step of one sentence is matched against all time-steps of the other sentence from multiple perspectives. Then, another BiLSTM layer is utilized to aggregate the matching results into a fix-length matching vector. Finally, based on the matching vector, a decision is made through a fully connected layer. We evaluate our model on three tasks: paraphrase identification, natural language inference and answer sentence selection. Experimental results on standard benchmark datasets show that our model achieves the state-of-the-art performance on all tasks.

scholarly journals Semantics-Aware BERT for Language Understanding

2020 ◽  
Vol 34 (05) ◽  
pp. 9628-9635
Author(s):  
Zhuosheng Zhang ◽  
Yuwei Wu ◽  
Hai Zhao ◽  
Zuchao Li ◽  
Shuailiang Zhang ◽  
...  
Keyword(s):  
Reading Comprehension ◽  
Natural Language ◽  
Language Model ◽  
Fine Tuning ◽  
Semantic Role Labeling ◽  
Language Understanding ◽  
Context Sensitive ◽  
Language Representation ◽  
Model Training ◽  
Machine Reading

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

scholarly journals Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

2020 ◽  
Vol 8 ◽  
pp. 264-280
Author(s):  
Sascha Rothe ◽  
Shashi Narayan ◽  
Aliaksei Severyn
Keyword(s):  
Natural Language Processing ◽  
Empirical Study ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Text Summarization ◽  
Neural Models ◽  
Language Understanding ◽  
Sequence Generation ◽  
Compute Time

Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2, and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.

scholarly journals Attention as Relation: Learning Supervised Multi-head Self-Attention for Relation Extraction

2020 ◽  
Author(s):  
Jie Liu ◽  
Shaowei Chen ◽  
Bingquan Wang ◽  
Jiaxin Zhang ◽  
Na Li ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Relation Extraction ◽  
Attention Mechanism ◽  
Entity Extraction ◽  
Relation Type ◽  
Benchmark Datasets ◽  
Relation Learning

Joint entity and relation extraction is critical for many natural language processing (NLP) tasks, which has attracted increasing research interest. However, it is still faced with the challenges of identifying the overlapping relation triplets along with the entire entity boundary and detecting the multi-type relations. In this paper, we propose an attention-based joint model, which mainly contains an entity extraction module and a relation detection module, to address the challenges. The key of our model is devising a supervised multi-head self-attention mechanism as the relation detection module to learn the token-level correlation for each relation type separately. With the attention mechanism, our model can effectively identify overlapping relations and flexibly predict the relation type with its corresponding intensity. To verify the effectiveness of our model, we conduct comprehensive experiments on two benchmark datasets. The experimental results demonstrate that our model achieves state-of-the-art performances.

MRN: Moment Relation Network for Natural Language Video Localization with Transfer Learning

2021 ◽  
pp. 2152009
Author(s):  
Siyu Jiang ◽  
Guobin Wu
Keyword(s):  
Probability Distribution ◽  
Natural Language ◽  
Transfer Learning ◽  
State Of The Art ◽  
Description Language ◽  
Large Margin ◽  
Shared Space ◽  
Learning Techniques ◽  
Benchmark Datasets ◽  
Temporal Segment

In this paper, we tackle the task of natural language video localization (NLVL): given an untrimmed video and a description language query, the goal is to localize the temporal segment within the video that best describes the natural language description. NLVL is challenging at the intersection of language and video understanding because a video may contain multiple segments of interests and the language may describe complicated temporal dependencies. Though existing approaches have achieved good performance, most of them did not fully consider the inherent differences between language and video modalities. Here, we propose Moment Relation Network (MRN) to reduce the divergence of the probability distribution of these two modalities. Specifically, MRN trains video and language subnets, and then uses transfer learning techniques to map the extracted features into an embedding-shared space where we calculate the similarity of two modalities using Mahalanobis distance metric, which is used to localize moments. Extensive experiments on benchmark datasets show that the proposed MRN significantly outperforms the state-of-the-art under the widely used metrics by a large margin.

Unsupervised extractive multi-document summarization method based on transfer learning from BERT multi-task fine-tuning

2021 ◽  
pp. 016555152199061
Author(s):  
Salima Lamsiyah ◽  
Abdelkader El Mahdaouy ◽  
Saïd El Alaoui Ouatik ◽  
Bernard Espinasse
Keyword(s):  
Transfer Learning ◽  
State Of The Art ◽  
Representation Learning ◽  
Fine Tuning ◽  
Text Representation ◽  
Document Summarization ◽  
Semantic Relationships ◽  
Benchmark Datasets ◽  
Tuning Methods ◽  
Fine Tune

Text representation is a fundamental cornerstone that impacts the effectiveness of several text summarization methods. Transfer learning using pre-trained word embedding models has shown promising results. However, most of these representations do not consider the order and the semantic relationships between words in a sentence, and thus they do not carry the meaning of a full sentence. To overcome this issue, the current study proposes an unsupervised method for extractive multi-document summarization based on transfer learning from BERT sentence embedding model. Moreover, to improve sentence representation learning, we fine-tune BERT model on supervised intermediate tasks from GLUE benchmark datasets using single-task and multi-task fine-tuning methods. Experiments are performed on the standard DUC’2002–2004 datasets. The obtained results show that our method has significantly outperformed several baseline methods and achieves a comparable and sometimes better performance than the recent state-of-the-art deep learning–based methods. Furthermore, the results show that fine-tuning BERT using multi-task learning has considerably improved the performance.

Sign in / Sign up

Export Citation Format

Share Document