scholarly journals Augmenting Paraphrase Generation with Syntax Information Using Graph Convolutional Networks

Entropy ◽  
2021 ◽  
Vol 23 (5) ◽  
pp. 566
Author(s):  
Xiaoqiang Chi ◽  
Yang Xiang
Keyword(s):  
Language Processing ◽  
Neural Nets ◽  
Linguistic Knowledge ◽  
Convolutional Network ◽  
Tree Structures ◽  
Convolutional Networks ◽  
Sequence Modeling ◽  
Syntactic Information ◽  
Dependency Trees ◽  
Paraphrase Generation

Paraphrase generation is an important yet challenging task in natural language processing. Neural network-based approaches have achieved remarkable success in sequence-to-sequence learning. Previous paraphrase generation work generally ignores syntactic information regardless of its availability, with the assumption that neural nets could learn such linguistic knowledge implicitly. In this work, we make an endeavor to probe into the efficacy of explicit syntactic information for the task of paraphrase generation. Syntactic information can appear in the form of dependency trees, which could be easily acquired from off-the-shelf syntactic parsers. Such tree structures could be conveniently encoded via graph convolutional networks to obtain more meaningful sentence representations, which could improve generated paraphrases. Through extensive experiments on four paraphrase datasets with different sizes and genres, we demonstrate the utility of syntactic information in neural paraphrase generation under the framework of sequence-to-sequence modeling. Specifically, our graph convolutional network-enhanced models consistently outperform their syntax-agnostic counterparts using multiple evaluation metrics.

scholarly journals Augmenting Paraphrase Generation with Syntax Information using Graph Convolutional Networks

2021 ◽  
Author(s):  
Xiaoqiang Chi ◽  
Yang Xiang
Keyword(s):  
Neural Network ◽  
Neural Nets ◽  
Linguistic Knowledge ◽  
Evaluation Metrics ◽  
Tree Structures ◽  
Convolutional Networks ◽  
Syntactic Information ◽  
Dependency Trees ◽  
Meaningful Sentence ◽  
Paraphrase Generation

Paraphrase generation is an important yet challenging task in NLP. Neural network-based approaches have achieved remarkable success in sequence-to-sequence(seq2seq) learning. Previous paraphrase generation work generally ignores syntactic information regardless of its availability, with the assumption that neural nets could learn such linguistic knowledge implicitly. In this work we make an endeavor to probe into the efficacy of explicit syntactic information for the task of paraphrase generation. Syntactic information can appear in the form of dependency trees which could be easily acquired from off-the-shelf syntactic parsers. Such tree structures could be conveniently encoded via graph convolutional networks(GCNs) to obtain more meaningful sentence representations, which could improve generated paraphrases. Through extensive experiments on four paraphrase datasets with different sizes and genres, we demonstrate the utility of syntactic information in neural paraphrase generation under the framework of seq2seq modeling. Specifically, our GCN-enhanced models consistently outperform their syntax-agnostic counterparts in multiple evaluation metrics.

scholarly journals Fusing Part-of-Speech Information in Low-Resource Neural Paraphrase Generation

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Xiaoqiang Chi ◽  
Yang Xiang
Keyword(s):  
Language Processing ◽  
Sequence Learning ◽  
Neural Nets ◽  
Linguistic Knowledge ◽  
Underlying Assumption ◽  
Low Resource ◽  
Part Of Speech ◽  
Multiple Datasets ◽  
Speech Information ◽  
Paraphrase Generation

Paraphrase generation is an essential yet challenging task in natural language processing. Neural-network-based approaches towards paraphrase generation have achieved remarkable success in recent years. Previous neural paraphrase generation approaches ignore linguistic knowledge, such as part-of-speech information regardless of its availability. The underlying assumption is that neural nets could learn such information implicitly when given sufficient data. However, it would be difficult for neural nets to learn such information properly when data are scarce. In this work, we endeavor to probe into the efficacy of explicit part-of-speech information for the task of paraphrase generation in low-resource scenarios. To this end, we devise three mechanisms to fuse part-of-speech information under the framework of sequence-to-sequence learning. We demonstrate the utility of part-of-speech information in low-resource paraphrase generation through extensive experiments on multiple datasets of varying sizes and genres.

scholarly journals Sentence Compression Using BERT and Graph Convolutional Networks

2021 ◽  
Vol 11 (21) ◽  
pp. 9910
Author(s):  
Yo-Han Park ◽  
Gyong-Ho Lee ◽  
Yong-Seok Choi ◽  
Kong-Joo Lee
Keyword(s):  
Language Processing ◽  
Large Scale ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Compression Model ◽  
Sentence Compression ◽  
Dependency Tree ◽  
Proposed Model ◽  
Syntactic Information ◽  
Input Sentence

Sentence compression is a natural language-processing task that produces a short paraphrase of an input sentence by deleting words from the input sentence while ensuring grammatical correctness and preserving meaningful core information. This study introduces a graph convolutional network (GCN) into a sentence compression task to encode syntactic information, such as dependency trees. As we upgrade the GCN to activate a directed edge, the compression model with the GCN layers can distinguish between parent and child nodes in a dependency tree when aggregating adjacent nodes. Furthermore, by increasing the number of GCN layers, the model can gradually collect high-order information of a dependency tree when propagating node information through the layers. We implement a sentence compression model for Korean and English, respectively. This model consists of three components: pre-trained BERT model, GCN layers, and a scoring layer. The scoring layer can determine whether a word should remain in a compressed sentence by relying on the word vector containing contextual and syntactic information encoded by BERT and GCN layers. To train and evaluate the proposed model, we used the Google sentence compression dataset for English and a Korean sentence compression corpus containing about 140,000 sentence pairs for Korean. The experimental results demonstrate that the proposed model achieves state-of-the-art performance for English. To the best of our knowledge, this sentence compression model based on the deep learning model trained with a large-scale corpus is the first attempt for Korean.

Unsupervised dependency parsing without training

2012 ◽  
Vol 18 (2) ◽  
pp. 187-203 ◽  
Author(s):  
ANDERS SØGAARD
Keyword(s):  
Language Processing ◽  
Training Data ◽  
Linguistic Knowledge ◽  
Dependency Parsing ◽  
Empirical Methods ◽  
Grammar Induction ◽  
Dependency Structure ◽  
Dependency Trees ◽  
Dependency Model ◽  
Parsing Algorithm

AbstractUsually unsupervised dependency parsers try to optimize the probability of a corpus by revising the dependency model that is assumed to have generated the corpus. In this paper we explore a different view in which a dependency structure is, among other things, a partial order on the nodes in terms of centrality or saliency. Under this assumption we directly model centrality and derive dependency trees from the ordering of words. The result is an approach to unsupervised dependency parsing that is very different from standard ones in that it requires no training data. The input words are ordered by centrality, and a parse is derived from the ranking using a simple deterministic parsing algorithm, relying on the universal dependency rules defined by Naseem et al. (Naseem, T., Chen, H., Barzilay, R., Johnson, M. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of Empirical Methods in Natural Language Processing, Boston, MA, USA, pp. 1234–44.). Our approach is evaluated on data from twelve different languages and is remarkably competitive.

scholarly journals A Graph Convolutional Network–Based Method for Chemical-Protein Interaction Extraction: Algorithm Development

10.2196/17643 ◽  
2020 ◽  
Vol 8 (5) ◽  
pp. e17643
Author(s):  
Erniu Wang ◽  
Fan Wang ◽  
Zhihao Yang ◽  
Lei Wang ◽  
Yin Zhang ◽  
...  
Keyword(s):  
Language Processing ◽  
Protein Interaction ◽  
State Of The Art ◽  
Significance Test ◽  
Biomedical Literature ◽  
Convolutional Network ◽  
Data Set ◽  
Dependency Structure ◽  
Syntactic Information ◽  
Biomedical Texts

Background Extracting the interactions between chemicals and proteins from the biomedical literature is important for many biomedical tasks such as drug discovery, medicine precision, and knowledge graph construction. Several computational methods have been proposed for automatic chemical-protein interaction (CPI) extraction. However, the majority of these proposed models cannot effectively learn semantic and syntactic information from complex sentences in biomedical texts. Objective To relieve this problem, we propose a method to effectively encode syntactic information from long text for CPI extraction. Methods Since syntactic information can be captured from dependency graphs, graph convolutional networks (GCNs) have recently drawn increasing attention in natural language processing. To investigate the performance of a GCN on CPI extraction, this paper proposes a novel GCN-based model. The model can effectively capture sequential information and long-range syntactic relations between words by using the dependency structure of input sentences. Results We evaluated our model on the ChemProt corpus released by BioCreative VI; it achieved an F-score of 65.17%, which is 1.07% higher than that of the state-of-the-art system proposed by Peng et al. As indicated by the significance test (P<.001), the improvement is significant. It indicates that our model is effective in extracting CPIs. The GCN-based model can better capture the semantic and syntactic information of the sentence compared to other models, therefore alleviating the problems associated with the complexity of biomedical literature. Conclusions Our model can obtain more information from the dependency graph than previously proposed models. Experimental results suggest that it is competitive to state-of-the-art methods and significantly outperforms other methods on the ChemProt corpus, which is the benchmark data set for CPI extraction.

scholarly journals Attention-Enhanced Graph Convolutional Networks for Aspect-Based Sentiment Classification with Multi-Head Attention

2021 ◽  
Vol 11 (8) ◽  
pp. 3640
Author(s):  
Guangtao Xu ◽  
Peiyu Liu ◽  
Zhenfang Zhu ◽  
Jie Liu ◽  
Fuyong Xu
Keyword(s):  
Sentiment Classification ◽  
Experimental Results ◽  
Sentence Structure ◽  
Convolutional Network ◽  
Structure Information ◽  
Convolutional Networks ◽  
Dependency Tree ◽  
Syntactic Information ◽  
Benchmark Datasets ◽  
Reasonable Use

The purpose of aspect-based sentiment classification is to identify the sentiment polarity of each aspect in a sentence. Recently, due to the introduction of Graph Convolutional Networks (GCN), more and more studies have used sentence structure information to establish the connection between aspects and opinion words. However, the accuracy of these methods is limited by noise information and dependency tree parsing performance. To solve this problem, we proposed an attention-enhanced graph convolutional network (AEGCN) for aspect-based sentiment classification with multi-head attention (MHA). Our proposed method can better combine semantic and syntactic information by introducing MHA and GCN. We also added an attention mechanism to GCN to enhance its performance. In order to verify the effectiveness of our proposed method, we conducted a lot of experiments on five benchmark datasets. The experimental results show that our proposed method can make more reasonable use of semantic and syntactic information, and further improve the performance of GCN.

scholarly journals Targeted Sentiment Classification Based on Attentional Encoding and Graph Convolutional Networks

2020 ◽  
Vol 10 (3) ◽  
pp. 957 ◽  
Author(s):  
Luwei Xiao ◽  
Xiaohui Hu ◽  
Yinong Chen ◽  
Yun Xue ◽  
Donghong Gu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Semantic Information ◽  
Sentiment Classification ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Dependency Tree ◽  
Proposed Model ◽  
Syntactic Information ◽  
Specific Goal ◽  
State Of Art

Targeted sentiment classification aims to predict the emotional trend of a specific goal. Currently, most methods (e.g., recurrent neural networks and convolutional neural networks combined with an attention mechanism) are not able to fully capture the semantic information of the context and they also lack a mechanism to explain the relevant syntactical constraints and long-range word dependencies. Therefore, syntactically irrelevant context words may mistakenly be recognized as clues to predict the target sentiment. To tackle these problems, this paper considers that the semantic information, syntactic information, and their interaction information are very crucial to targeted sentiment analysis, and propose an attentional-encoding-based graph convolutional network (AEGCN) model. Our proposed model is mainly composed of multi-head attention and an improved graph convolutional network built over the dependency tree of a sentence. Pre-trained BERT is applied to this task, and new state-of-art performance is achieved. Experiments on five datasets show the effectiveness of the model proposed in this paper compared with a series of the latest models.

scholarly journals A Graph Convolutional Network–Based Method for Chemical-Protein Interaction Extraction: Algorithm Development (Preprint)

2019 ◽  
Author(s):  
Erniu Wang ◽  
Fan Wang ◽  
Zhihao Yang ◽  
Lei Wang ◽  
Yin Zhang ◽  
...  
Keyword(s):  
Language Processing ◽  
Protein Interaction ◽  
State Of The Art ◽  
Significance Test ◽  
Biomedical Literature ◽  
Convolutional Network ◽  
Data Set ◽  
Dependency Structure ◽  
Syntactic Information ◽  
Biomedical Texts

BACKGROUND Extracting the interactions between chemicals and proteins from the biomedical literature is important for many biomedical tasks such as drug discovery, medicine precision, and knowledge graph construction. Several computational methods have been proposed for automatic chemical-protein interaction (CPI) extraction. However, the majority of these proposed models cannot effectively learn semantic and syntactic information from complex sentences in biomedical texts. OBJECTIVE To relieve this problem, we propose a method to effectively encode syntactic information from long text for CPI extraction. METHODS Since syntactic information can be captured from dependency graphs, graph convolutional networks (GCNs) have recently drawn increasing attention in natural language processing. To investigate the performance of a GCN on CPI extraction, this paper proposes a novel GCN-based model. The model can effectively capture sequential information and long-range syntactic relations between words by using the dependency structure of input sentences. RESULTS We evaluated our model on the ChemProt corpus released by BioCreative VI; it achieved an F-score of 65.17%, which is 1.07% higher than that of the state-of-the-art system proposed by Peng et al. As indicated by the significance test (<i>P</i>&lt;.001), the improvement is significant. It indicates that our model is effective in extracting CPIs. The GCN-based model can better capture the semantic and syntactic information of the sentence compared to other models, therefore alleviating the problems associated with the complexity of biomedical literature. CONCLUSIONS Our model can obtain more information from the dependency graph than previously proposed models. Experimental results suggest that it is competitive to state-of-the-art methods and significantly outperforms other methods on the ChemProt corpus, which is the benchmark data set for CPI extraction.

scholarly journals Graph Convolutional Networks with Bidirectional Attention for Aspect-Based Sentiment Classification

2021 ◽  
Vol 11 (4) ◽  
pp. 1528
Author(s):  
Jie Liu ◽  
Peiyu Liu ◽  
Zhenfang Zhu ◽  
Xiaowen Li ◽  
Guangtao Xu
Keyword(s):  
Attention Mechanism ◽  
Sentiment Classification ◽  
Long Distance ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Syntactic Information ◽  
Position Coding ◽  
Novel Method

Aspect-based sentiment classification aims at determining the corresponding sentiment of a particular aspect. Many sophisticated approaches, such as attention mechanisms and Graph Convolutional Networks, have been widely used to address this challenge. However, most of the previous methods have not well analyzed the role of words and long-distance dependencies, and the interaction between context and aspect terms is not well realized, which greatly limits the effectiveness of the model. In this paper, we propose an effective and novel method using attention mechanism and graph convolutional network (ATGCN). Firstly, we make full use of multi-head attention and point-wise convolution transformation to obtain the hidden state. Secondly, we introduce position coding in the model, and use Graph Convolutional Networks to obtain syntactic information and long-distance dependencies. Finally, the interaction between context and aspect terms is further realized by bidirectional attention. Experiments on three benchmarking collections indicate the effectiveness of ATGCN.

Putting up barriers

2017 ◽  
Author(s):  
David J. Lobina
Keyword(s):  
Knowledge Base ◽  
Real Time ◽  
Language Processing ◽  
Cognitive Domain ◽  
Linguistic Knowledge ◽  
Levels Of Explanation ◽  
Real Time Processing ◽  
Time Processing ◽  
Neural Underpinnings ◽  
Different Levels

The study of cognitive phenomena is best approached in an orderly manner. It must begin with an analysis of the function in intension at the heart of any cognitive domain (its knowledge base), then proceed to the manner in which such knowledge is put into use in real-time processing, concluding with a domain’s neural underpinnings, its development in ontogeny, etc. Such an approach to the study of cognition involves the adoption of different levels of explanation/description, as prescribed by David Marr and many others, each level requiring its own methodology and supplying its own data to be accounted for. The study of recursion in cognition is badly in need of a systematic and well-ordered approach, and this chapter lays out the blueprint to be followed in the book by focusing on a strict separation between how this notion applies in linguistic knowledge and how it manifests itself in language processing.

Sign in / Sign up

Export Citation Format

Share Document