Natural language modeling with syntactic structure dependency

2020 ◽  
Vol 523 ◽  
pp. 220-233
Author(s):  
Kai Shuang ◽  
Yijia Tan ◽  
Zhun Cai ◽  
Yue Sun
2021 ◽  
Author(s):  
Abdul Wahab ◽  
Rafet Sifa

<div> <div> <div> <p> </p><div> <div> <div> <p>In this paper, we propose a new model named DIBERT which stands for Dependency Injected Bidirectional Encoder Representations from Transformers. DIBERT is a variation of the BERT and has an additional third objective called Parent Prediction (PP) apart from Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). PP injects the syntactic structure of a dependency tree while pre-training the DIBERT which generates syntax-aware generic representations. We use the WikiText-103 benchmark dataset to pre-train both BERT- Base and DIBERT. After fine-tuning, we observe that DIBERT performs better than BERT-Base on various downstream tasks including Semantic Similarity, Natural Language Inference and Sentiment Analysis. </p> </div> </div> </div> </div> </div> </div>


2021 ◽  
Author(s):  
Abdul Wahab ◽  
Rafet Sifa

<div> <div> <div> <p> </p><div> <div> <div> <p>In this paper, we propose a new model named DIBERT which stands for Dependency Injected Bidirectional Encoder Representations from Transformers. DIBERT is a variation of the BERT and has an additional third objective called Parent Prediction (PP) apart from Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). PP injects the syntactic structure of a dependency tree while pre-training the DIBERT which generates syntax-aware generic representations. We use the WikiText-103 benchmark dataset to pre-train both BERT- Base and DIBERT. After fine-tuning, we observe that DIBERT performs better than BERT-Base on various downstream tasks including Semantic Similarity, Natural Language Inference and Sentiment Analysis. </p> </div> </div> </div> </div> </div> </div>


2021 ◽  
Author(s):  
Abdul Wahab ◽  
Rafet Sifa

<div> <div> <div> <p> </p><div> <div> <div> <p>In this paper, we propose a new model named DIBERT which stands for Dependency Injected Bidirectional Encoder Representations from Transformers. DIBERT is a variation of the BERT and has an additional third objective called Parent Prediction (PP) apart from Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). PP injects the syntactic structure of a dependency tree while pre-training the DIBERT which generates syntax-aware generic representations. We use the WikiText-103 benchmark dataset to pre-train both BERT- Base and DIBERT. After fine-tuning, we observe that DIBERT performs better than BERT-Base on various downstream tasks including Semantic Similarity, Natural Language Inference and Sentiment Analysis. </p> </div> </div> </div> </div> </div> </div>


2021 ◽  
Vol 11 (7) ◽  
pp. 3095
Author(s):  
Suhyune Son ◽  
Seonjeong Hwang ◽  
Sohyeun Bae ◽  
Soo Jun Park ◽  
Jang-Hwan Choi

Multi-task learning (MTL) approaches are actively used for various natural language processing (NLP) tasks. The Multi-Task Deep Neural Network (MT-DNN) has contributed significantly to improving the performance of natural language understanding (NLU) tasks. However, one drawback is that confusion about the language representation of various tasks arises during the training of the MT-DNN model. Inspired by the internal-transfer weighting of MTL in medical imaging, we introduce a Sequential and Intensive Weighted Language Modeling (SIWLM) scheme. The SIWLM consists of two stages: (1) Sequential weighted learning (SWL), which trains a model to learn entire tasks sequentially and concentrically, and (2) Intensive weighted learning (IWL), which enables the model to focus on the central task. We apply this scheme to the MT-DNN model and call this model the MTDNN-SIWLM. Our model achieves higher performance than the existing reference algorithms on six out of the eight GLUE benchmark tasks. Moreover, our model outperforms MT-DNN by 0.77 on average on the overall task. Finally, we conducted a thorough empirical investigation to determine the optimal weight for each GLUE task.


2022 ◽  
pp. 1-13
Author(s):  
Denis Paperno

Abstract Can recurrent neural nets, inspired by human sequential data processing, learn to understand language? We construct simplified datasets reflecting core properties of natural language as modeled in formal syntax and semantics: recursive syntactic structure and compositionality. We find LSTM and GRU networks to generalise to compositional interpretation well, but only in the most favorable learning settings, with a well-paced curriculum, extensive training data, and left-to-right (but not right-to-left) composition.


Author(s):  
Nia Shafira ◽  
◽  
Etin Martiana ◽  
Rengga Asmara

As the main train service provider company in Indonesia, PT Kereta Api Indonesia (PT KAI) has many customers who need information. In order to maintain customer loyalty, PT KAI must respond quickly and be adaptive to technology to provide the best service to customers. Limited human resources make PT KAI unable to serve customers simultaneously, so customers often have to wait for a response. In order to provide the best service, automatic messages are needed in order to help customer service performance respond quickly and at the same time with no cost, access anytime and anywhere. This study proposes a new approach with chatbots as a medium for conveying automatic information quickly and simultaneously. This chatbot is made with a computational language that focuses on natural language modeling and cosine similarity as a method for calculating the proximity of inputs and databases. This research can help PT KAI's customer service workers to answer customer needs automatically.


1979 ◽  
Vol 15 (1) ◽  
pp. 39-47 ◽  
Author(s):  
Geoffrey Sampson

Many contemporary linguists hold that an adequate description of a natural language must represent many of its vocabulary items as syntactically and/or semantically complex. A sentence containing the word kill, for instance, will on this view be assigned a ‘deep syntactic structure’ or ‘semantic representation’ in which kill is represented by a portion or portions of tree-structure, the lowest nodes of which are labelled with ‘semantic primitives’ such as CAUSE and DIE, or CAUSE, BECOME, NOT and ALIVE. In the case of words such as cats or walked, which are formed in accordance with productive rules of ‘inflexional’ rather than ‘derivational’ morphology, there is little dispute that their composite status will be reflected at most or all levels of linguistic representation. (That is why I refer, above, to ‘vocabulary items’: cat and cats may be called different ‘words’, but not different elements of the English vocbulary.) When morphologically simple words such as kill are treated as composite at a ‘deeper’ level, I, for one, find my credulity strained to breaking point. (The case of words formed in accordance with productive or non-productive rules of derivational morphology, such as killer or kingly, is an intermediate one and I shall briefly return to it below.)


2008 ◽  
Vol 34 (4) ◽  
pp. 597-614 ◽  
Author(s):  
Trevor Cohn ◽  
Chris Callison-Burch ◽  
Mirella Lapata

Automatic paraphrasing is an important component in many natural language processing tasks. In this article we present a new parallel corpus with paraphrase annotations. We adopt a definition of paraphrase based on word alignments and show that it yields high inter-annotator agreement. As Kappa is suited to nominal data, we employ an alternative agreement statistic which is appropriate for structured alignment tasks. We discuss how the corpus can be usefully employed in evaluating paraphrase systems automatically (e.g., by measuring precision, recall, and F1) and also in developing linguistically rich paraphrase models based on syntactic structure.


Sign in / Sign up

Export Citation Format

Share Document