SANTM: Efficient Self-attention-driven Network for Text Matching

2022 ◽  
Vol 22 (3) ◽  
pp. 1-21
Author(s):  
Prayag Tiwari ◽  
Amit Kumar Jaiswal ◽  
Sahil Garg ◽  
Ilsun You
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
The State ◽  
Attention Mechanism ◽  
Matching Problems ◽  
Attention Model ◽  
Extra Information ◽  
Textual Entailment ◽  
Benchmark Datasets ◽  
Text Matching

Self-attention mechanisms have recently been embraced for a broad range of text-matching applications. Self-attention model takes only one sentence as an input with no extra information, i.e., one can utilize the final hidden state or pooling. However, text-matching problems can be interpreted either in symmetrical or asymmetrical scopes. For instance, paraphrase detection is an asymmetrical task, while textual entailment classification and question-answer matching are considered asymmetrical tasks. In this article, we leverage attractive properties of self-attention mechanism and proposes an attention-based network that incorporates three key components for inter-sequence attention: global pointwise features, preceding attentive features, and contextual features while updating the rest of the components. Our model follows evaluation on two benchmark datasets cover tasks of textual entailment and question-answer matching. The proposed efficient Self-attention-driven Network for Text Matching outperforms the state of the art on the Stanford Natural Language Inference and WikiQA datasets with much fewer parameters.

scholarly journals Large-scale Semantic Parsing without Question-Answer Pairs

2014 ◽  
Vol 2 ◽  
pp. 377-392 ◽  
Author(s):  
Siva Reddy ◽  
Mirella Lapata ◽  
Mark Steedman
Keyword(s):  
Natural Language ◽  
Large Scale ◽  
Graph Matching ◽  
State Of The Art ◽  
The State ◽  
Semantic Parsing ◽  
Matching Problem ◽  
Weak Supervision ◽  
Benchmark Datasets

In this paper we introduce a novel semantic parsing approach to query Freebase in natural language without requiring manual annotations or question-answer pairs. Our key insight is to represent natural language via semantic graphs whose topology shares many commonalities with Freebase. Given this representation, we conceptualize semantic parsing as a graph matching problem. Our model converts sentences to semantic graphs using CCG and subsequently grounds them to Freebase guided by denotations as a form of weak supervision. Evaluation experiments on a subset of the Free917 and WebQuestions benchmark datasets show our semantic parser improves over the state of the art.

scholarly journals A Span-based Joint Model for Opinion Target Extraction and Target Sentiment Classification

2019 ◽  
Author(s):  
Yan Zhou ◽  
Longtao Huang ◽  
Tao Guo ◽  
Jizhong Han ◽  
Songlin Hu
Keyword(s):  
Sentiment Analysis ◽  
State Of The Art ◽  
Joint Model ◽  
The State ◽  
Attention Mechanism ◽  
Sentiment Classification ◽  
Global Information ◽  
Target Extraction ◽  
Benchmark Datasets ◽  
Tagging Methods

Target-Based Sentiment Analysis aims at extracting opinion targets and classifying the sentiment polarities expressed on each target. Recently, token based sequence tagging methods have been successfully applied to jointly solve the two tasks, which aims to predict a tag for each token. Since they do not treat a target containing several words as a whole, it might be difficult to make use of the global information to identify that opinion target, leading to incorrect extraction. Independently predicting the sentiment for each token may also lead to sentiment inconsistency for different words in an opinion target. In this paper, inspired by span-based methods in NLP, we propose a simple and effective joint model to conduct extraction and classification at span level rather than token level. Our model first emulates spans with one or more tokens and learns their representation based on the tokens inside. And then, a span-aware attention mechanism is designed to compute the sentiment information towards each span. Extensive experiments on three benchmark datasets show that our model consistently outperforms the state-of-the-art methods.

scholarly journals Bilateral Multi-Perspective Matching for Natural Language Sentences

2017 ◽  
Author(s):  
Zhiguo Wang ◽  
Wael Hamza ◽  
Radu Florian
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
The State ◽  
Experimental Results ◽  
The Other ◽  
Multiple Perspectives ◽  
Time Step ◽  
Benchmark Datasets ◽  
Sentence Matching ◽  
Fully Connected

Natural language sentence matching is a fundamental technology for a variety of tasks. Previous approaches either match sentences from a single direction or only apply single granular (word-by-word or sentence-by-sentence) matching. In this work, we propose a bilateral multi-perspective matching (BiMPM) model. Given two sentences P and Q, our model first encodes them with a BiLSTM encoder. Next, we match the two encoded sentences in two directions P against Q and P against Q. In each matching direction, each time step of one sentence is matched against all time-steps of the other sentence from multiple perspectives. Then, another BiLSTM layer is utilized to aggregate the matching results into a fix-length matching vector. Finally, based on the matching vector, a decision is made through a fully connected layer. We evaluate our model on three tasks: paraphrase identification, natural language inference and answer sentence selection. Experimental results on standard benchmark datasets show that our model achieves the state-of-the-art performance on all tasks.

scholarly journals Attention as Relation: Learning Supervised Multi-head Self-Attention for Relation Extraction

2020 ◽  
Author(s):  
Jie Liu ◽  
Shaowei Chen ◽  
Bingquan Wang ◽  
Jiaxin Zhang ◽  
Na Li ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Relation Extraction ◽  
Attention Mechanism ◽  
Entity Extraction ◽  
Relation Type ◽  
Benchmark Datasets ◽  
Relation Learning

Joint entity and relation extraction is critical for many natural language processing (NLP) tasks, which has attracted increasing research interest. However, it is still faced with the challenges of identifying the overlapping relation triplets along with the entire entity boundary and detecting the multi-type relations. In this paper, we propose an attention-based joint model, which mainly contains an entity extraction module and a relation detection module, to address the challenges. The key of our model is devising a supervised multi-head self-attention mechanism as the relation detection module to learn the token-level correlation for each relation type separately. With the attention mechanism, our model can effectively identify overlapping relations and flexibly predict the relation type with its corresponding intensity. To verify the effectiveness of our model, we conduct comprehensive experiments on two benchmark datasets. The experimental results demonstrate that our model achieves state-of-the-art performances.

scholarly journals Semantic Sentence Matching with Densely-Connected Recurrent and Co-Attentive Information

2019 ◽  
Vol 33 ◽  
pp. 6586-6593 ◽  
Author(s):  
Seonhoon Kim ◽  
Inho Kang ◽  
Nojun Kwak
Keyword(s):  
Neural Network ◽  
Natural Language ◽  
Question Answering ◽  
State Of The Art ◽  
Attention Mechanism ◽  
Semantic Relationship ◽  
Convolutional Network ◽  
Benchmark Datasets ◽  
Feature Information ◽  
Sentence Matching

Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.

scholarly journals Weakly-Supervised Video Re-Localization with Multiscale Attention Model

2020 ◽  
Vol 34 (07) ◽  
pp. 11077-11084
Author(s):  
Yung-Han Huang ◽  
Kuang-Jui Hsu ◽  
Shyh-Kang Jeng ◽  
Yen-Yu Lin
Keyword(s):  
Feature Extraction ◽  
State Of The Art ◽  
The State ◽  
Attention Mechanism ◽  
Attention Model ◽  
Temporal Features ◽  
Target Segment ◽  
Public Dataset ◽  
Art Performance ◽  
Weakly Supervised

Video re-localization aims to localize a sub-sequence, called target segment, in an untrimmed reference video that is similar to a given query video. In this work, we propose an attention-based model to accomplish this task in a weakly supervised setting. Namely, we derive our CNN-based model without using the annotated locations of the target segments in reference videos. Our model contains three modules. First, it employs a pre-trained C3D network for feature extraction. Second, we design an attention mechanism to extract multiscale temporal features, which are then used to estimate the similarity between the query video and a reference video. Third, a localization layer detects where the target segment is in the reference video by determining whether each frame in the reference video is consistent with the query video. The resultant CNN model is derived based on the proposed co-attention loss which discriminatively separates the target segment from the reference video. This loss maximizes the similarity between the query video and the target segment while minimizing the similarity between the target segment and the rest of the reference video. Our model can be modified to fully supervised re-localization. Our method is evaluated on a public dataset and achieves the state-of-the-art performance under both weakly supervised and fully supervised settings.

scholarly journals Conversational Chatbot with Attention Model

2019 ◽  
Vol 9 (2) ◽  
pp. 3537-3540
Keyword(s):  
Artificial Intelligence ◽  
Natural Language ◽  
Machine Translation ◽  
State Of The Art ◽  
Attention Mechanism ◽  
Fine Tuning ◽  
Beam Search ◽  
Attention Model ◽  
Transformer Model ◽  
Efficient Sequence

A Chatbot is an Artificial Intelligence (AI) software that can give a simulation of a conversation between two humans. This Chatbot is based on State of the Art Transformer model architecture which works on Attention mechanism. The transformer model is a very efficient Sequence to Sequence model. Machine translation is at its core , simply a task in which you map the sentence to another sentence. Sentences consist of words that are equivalent to mapping to a different sequence. Beam search and Byte-pair encoding are the algorithms used in our model for heuristic searching in decoder units. A combination of many Unsupervised prediction tasks were carried out by fine-tuning using a multi-task objective every time the user starts the conversation. It takes a new persona for every new session opened and communicates with that persona which is chosen at random. Forwarding the perplexity by the ability to understand and generate natural language this model gives a whooping Hits@1 score efficiency as high as 80.9 percentage.

Single-shot Semantic Matching Network for Moment Localization in Videos

2021 ◽  
Vol 17 (3) ◽  
pp. 1-14
Author(s):  
Xinfang Liu ◽  
Xiushan Nie ◽  
Junya Teng ◽  
Li Lian ◽  
Yilong Yin
Keyword(s):  
Natural Language ◽  
Fixed Number ◽  
Single Shot ◽  
Semantic Matching ◽  
Matching Network ◽  
Attention Model ◽  
Semantic Relationships ◽  
Natural Language Query ◽  
Benchmark Datasets ◽  
Short Memory

Moment localization in videos using natural language refers to finding the most relevant segment from videos given a natural language query. Most of the existing methods require video segment candidates for further matching with the query, which leads to extra computational costs, and they may also not locate the relevant moments under any length evaluated. To address these issues, we present a lightweight single-shot semantic matching network (SSMN) to avoid the complex computations required to match the query and the segment candidates, and the proposed SSMN can locate moments of any length theoretically. Using the proposed SSMN, video features are first uniformly sampled to a fixed number, while the query sentence features are generated and enhanced by GloVe, long-term short memory (LSTM), and soft-attention modules. Subsequently, the video features and sentence features are fed to an enhanced cross-modal attention model to mine the semantic relationships between vision and language. Finally, a score predictor and a location predictor are designed to locate the start and stop indexes of the query moment. We evaluate the proposed method on two benchmark datasets and the experimental results demonstrate that SSMN outperforms state-of-the-art methods in both precision and efficiency.

Named Entity Recognition and Relation Extraction

2021 ◽  
Vol 54 (1) ◽  
pp. 1-39
Author(s):  
Zara Nasar ◽  
Syed Waqar Jaffry ◽  
Muhammad Kamran Malik
Keyword(s):  
Deep Learning ◽  
State Of The Art ◽  
Named Entity Recognition ◽  
Relation Extraction ◽  
The State ◽  
Entity Recognition ◽  
Joint Models ◽  
Named Entity ◽  
Textual Data ◽  
Benchmark Datasets

With the advent of Web 2.0, there exist many online platforms that result in massive textual-data production. With ever-increasing textual data at hand, it is of immense importance to extract information nuggets from this data. One approach towards effective harnessing of this unstructured textual data could be its transformation into structured text. Hence, this study aims to present an overview of approaches that can be applied to extract key insights from textual data in a structured way. For this, Named Entity Recognition and Relation Extraction are being majorly addressed in this review study. The former deals with identification of named entities, and the latter deals with problem of extracting relation between set of entities. This study covers early approaches as well as the developments made up till now using machine learning models. Survey findings conclude that deep-learning-based hybrid and joint models are currently governing the state-of-the-art. It is also observed that annotated benchmark datasets for various textual-data generators such as Twitter and other social forums are not available. This scarcity of dataset has resulted into relatively less progress in these domains. Additionally, the majority of the state-of-the-art techniques are offline and computationally expensive. Last, with increasing focus on deep-learning frameworks, there is need to understand and explain the under-going processes in deep architectures.

scholarly journals DRr-Net: Dynamic Re-Read Network for Sentence Semantic Matching

2019 ◽  
Vol 33 ◽  
pp. 7442-7449 ◽  
Author(s):  
Kun Zhang ◽  
Guangyi Lv ◽  
Linyuan Wang ◽  
Le Wu ◽  
Enhong Chen ◽  
...  
Keyword(s):  
Natural Language ◽  
Small Region ◽  
Psychological Research ◽  
Attention Mechanism ◽  
Semantic Relations ◽  
Semantic Matching ◽  
Close Attention ◽  
Original Sentence ◽  
Benchmark Datasets ◽  
Sentence Matching

Sentence semantic matching requires an agent to determine the semantic relation between two sentences, which is widely used in various natural language tasks such as Natural Language Inference (NLI) and Paraphrase Identification (PI). Among all matching methods, attention mechanism plays an important role in capturing the semantic relations and properly aligning the elements of two sentences. Previous methods utilized attention mechanism to select important parts of sentences at one time. However, the important parts of the sentence during semantic matching are dynamically changing with the degree of sentence understanding. Selecting the important parts at one time may be insufficient for semantic understanding. To this end, we propose a Dynamic Re-read Network (DRr-Net) approach for sentence semantic matching, which is able to pay close attention to a small region of sentences at each step and re-read the important words for better sentence semantic understanding. To be specific, we first employ Attention Stack-GRU (ASG) unit to model the original sentence repeatedly and preserve all the information from bottom-most word embedding input to up-most recurrent output. Second, we utilize Dynamic Re-read (DRr) unit to pay close attention to one important word at one time with the consideration of learned information and re-read the important words for better sentence semantic understanding. Extensive experiments on three sentence matching benchmark datasets demonstrate that DRr-Net has the ability to model sentence semantic more precisely and significantly improve the performance of sentence semantic matching. In addition, it is very interesting that some of finding in our experiments are consistent with the findings of psychological research.

Sign in / Sign up

Export Citation Format

Share Document