Korean Dependency Parsing using Token-Level Contextual Representation in Pre-trained Language Model

In recent years, the research on dependency parsing focuses on improving the accuracy of the domain-specific (in-domain) test datasets and has made remarkable progress. However, there are innumerable scenarios in the real world that are not covered by the dataset, namely, the out-of-domain dataset. As a result, parsers that perform well on the in-domain data usually suffer from significant performance degradation on the out-of-domain data. Therefore, to adapt the existing in-domain parsers with high performance to a new domain scenario, cross-domain transfer learning methods are essential to solve the domain problem in parsing. This paper examines two scenarios for cross-domain transfer learning: semi-supervised and unsupervised cross-domain transfer learning. Specifically, we adopt a pre-trained language model BERT for training on the source domain (in-domain) data at the subword level and introduce self-training methods varied from tri-training for these two scenarios. The evaluation results on the NLPCC-2019 shared task and universal dependency parsing task indicate the effectiveness of the adopted approaches on cross-domain transfer learning and show the potential of self-learning to cross-lingual transfer learning.

Download Full-text

Efficient Contextual Representation Learning With Continuous Outputs

Transactions of the Association for Computational Linguistics ◽

10.1162/tacl_a_00289 ◽

2019 ◽

Vol 7 ◽

pp. 611-624 ◽

Cited By ~ 1

Author(s):

Liunian Harold Li ◽

Patrick H. Chen ◽

Cho-Jui Hsieh ◽

Kai-Wei Chang

Keyword(s):

Target Word ◽

Language Processing ◽

Language Model ◽

Representation Learning ◽

Great Success ◽

Training Procedure ◽

Vocabulary Size ◽

Speed Advantage ◽

Contextual Representation ◽

High Computational Complexity

Contextual representation models have achieved great success in improving various downstream natural language processing tasks. However, these language-model-based encoders are difficult to train due to their large parameter size and high computational complexity. By carefully examining the training procedure, we observe that the softmax layer, which predicts a distribution of the target word, often induces significant overhead, especially when the vocabulary size is large. Therefore, we revisit the design of the output layer and consider directly predicting the pre-trained embedding of the target word for a given context. When applied to ELMo, the proposed approach achieves a 4-fold speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks. Further analysis shows that the approach maintains the speed advantage under various settings, even when the sentence encoder is scaled up.

Download Full-text

TAGGING AND PARSING OF MULTIDOMAIN COLLECTIONS

Computational Linguistics and Intellectual Technologies ◽

10.28995/2075-7182-2020-19-670-683 ◽

2020 ◽

Author(s):

A. A. Sorokin ◽

◽

I. M. Smurov ◽

D. P. Kirianov ◽

◽

...

Keyword(s):

Language Model ◽

Model Performance ◽

Fine Tuning ◽

Dependency Parsing ◽

Aggregate Measure

In this paper we describe our submission to GramEval2020 competition on morphological tagging, lemmatization and dependency parsing. Our model uses biaffine attention over the BERT representations. The main feature of our work is the extensive usage of language model, tagger and parser fine-tuning on several distinct genres and the implementation of genre classifier. To deal with dataset idiosyncrasies we also extensively apply handwritten rules. Our model took second place in the overall model performance scoring 90.8 aggregate measure over all 4 tasks

Download Full-text

LVBERT: Transformer-Based Model for Latvian Language Understanding

Frontiers in Artificial Intelligence and Applications - Human Language Technologies – The Baltic Perspective ◽

10.3233/faia200610 ◽

2020 ◽

Author(s):

Artūrs Znotiņš ◽

Guntis Barzdiņš

Keyword(s):

State Of The Art ◽

Language Model ◽

Named Entity Recognition ◽

Entity Recognition ◽

Future Research ◽

Dependency Parsing ◽

Named Entity ◽

Part Of Speech Tagging ◽

Part Of Speech ◽

Speech Tagging

This paper presents LVBERT – the first publicly available monolingual language model pre-trained for Latvian. We show that LVBERT improves the state-of-the-art for three Latvian NLP tasks including Part-of-Speech tagging, Named Entity Recognition and Universal Dependency parsing. We release LVBERT to facilitate future research and downstream applications for Latvian NLP.

Download Full-text

Query expansion based on global dependency parsing analysis

Information Science and Management Engineering ◽

10.2495/isme131952 ◽

2013 ◽

Author(s):

Qinyuan Xiang ◽

Weijiang Li ◽

Hui Deng ◽

Feng Wang

Keyword(s):

Query Expansion ◽

Dependency Parsing

Download Full-text

Interpretation of Nonsense as a Method of Children Imagination Development (as exemplified in Carroll L. «Through the Looking-Glass and What Alice Found There»)

Current Issues in Philology and Pedagogical Linguistics ◽

10.29025/2079-6021-2020-4-200-212 ◽

2020 ◽

pp. 200-212

Author(s):

Larisa V. Kalashnikova

Keyword(s):

Creative Thinking ◽

Language Model ◽

Word Formation ◽

The Third ◽

New Meanings ◽

The World ◽

Looking Glass ◽

Rules Of The Game ◽

Powerful Incentive ◽

Algorithmic Thinking

The article enlightens the probem of nonsense and its role in the development of creative thinking and fantasy, and the way how the interpretation of nonsense affects children imagination. The function of imagination inherent to a person, and especially to a child, has a powerful potential – to create artificially new metaphorical models, absurd and most incredible situations based on self-amazement. Children are able to measure the properties of unfamiliar objects with the properties of known things. It is not difficult for small researchers to replace incomprehensible meanings with familiar ones; to think over situations, to make analogies, to transfer signs and properties of one object to another. The problem of nonsense research is interesting and relevant. The element of the game is an integral component of nonsense. In the process of playing, children cognize the world, learn to interact with the world, imitating the adults behavior. Imagination and fantasy help the child to invent his own rules of the game, to choose language elements that best suit his ideas. The child uses the learned productive models of the language system to create their own models and their own language, attracting language signs: words, morphs, sentences. Children’s dictionary stimulates word formation and language nomination processes. Nonsense-words are the result of children’s dictionary, speech errors and occazional formations, presented in the form of contamination, phonetic transformations, lexical substitution, implemented on certain models. The first two models are phonetic imitation and hybrid speech, based on the natural language model. The third model of designing nonsense is represented by words that have no meaning at all and can be attributed to words-portmonaie. Due to the flexibility of interframe relationships and the lack of algorithmic thinking, children can not only capture the implicit similarity of objects and phenomena, but also create it through their imagination. Interpretation of nonsense is an effective method of developing imagination in children, because metaphors, nonsense as a means of creating new meanings, modeling new content from fragments of one’s own experience, are a powerful incentive for creative thinking.

Download Full-text