scholarly journals Transformers analyzing poetry: multilingual metrical pattern prediction with transfomer-based language models

2021 ◽  
Author(s):  
Javier de la Rosa ◽  
Álvaro Pérez ◽  
Mirella de Sisto ◽  
Laura Hernández ◽  
Aitor Díaz ◽  
...  
Keyword(s):  
Structural Information ◽  
Pattern Identification ◽  
Language Models ◽  
Distant Reading ◽  
Rhetorical Devices ◽  
Metrical Pattern ◽  
The Creation ◽  
Cross Lingual ◽  
Pattern Prediction

AbstractThe splitting of words into stressed and unstressed syllables is the foundation for the scansion of poetry, a process that aims at determining the metrical pattern of a line of verse within a poem. Intricate language rules and their exceptions, as well as poetic licenses exerted by the authors, make calculating these patterns a nontrivial task. Some rhetorical devices shrink the metrical length, while others might extend it. This opens the door for interpretation and further complicates the creation of automated scansion algorithms useful for automatically analyzing corpora on a distant reading fashion. In this paper, we compare the automated metrical pattern identification systems available for Spanish, English, and German, against fine-tuned monolingual and multilingual language models trained on the same task. Despite being initially conceived as models suitable for semantic tasks, our results suggest that transformers-based models retain enough structural information to perform reasonably well for Spanish on a monolingual setting, and outperforms both for English and German when using a model trained on the three languages, showing evidence of the benefits of cross-lingual transfer between the languages.

scholarly journals Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

2016 ◽  
Vol 4 ◽  
pp. 521-535 ◽  
Author(s):  
Tal Linzen ◽  
Emmanuel Dupoux ◽  
Yoav Goldberg
Keyword(s):  
Language Processing ◽  
Short Term Memory ◽  
Structural Information ◽  
Syntactic Structure ◽  
Language Modeling ◽  
Language Models ◽  
Grammatical Structure ◽  
Long Distance ◽  
Target Number ◽  
Statistical Regularities

The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture’s grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.

scholarly journals Unsupervised Neural Machine Translation with SMT as Posterior Regularization

2019 ◽  
Vol 33 ◽  
pp. 241-248 ◽  
Author(s):  
Shuo Ren ◽  
Zhirui Zhang ◽  
Shujie Liu ◽  
Ming Zhou ◽  
Shuai Ma
Keyword(s):  
Machine Translation ◽  
Language Models ◽  
Translation Process ◽  
Weak Supervision ◽  
Neural Machine Translation ◽  
Back Translation ◽  
Negative Effect ◽  
Model Training ◽  
Cross Lingual ◽  
Pseudo Data

Without real bilingual corpus available, unsupervised Neural Machine Translation (NMT) typically requires pseudo parallel data generated with the back-translation method for the model training. However, due to weak supervision, the pseudo data inevitably contain noises and errors that will be accumulated and reinforced in the subsequent training process, leading to bad translation performance. To address this issue, we introduce phrase based Statistic Machine Translation (SMT) models which are robust to noisy data, as posterior regularizations to guide the training of unsupervised NMT models in the iterative back-translation process. Our method starts from SMT models built with pre-trained language models and word-level translation tables inferred from cross-lingual embeddings. Then SMT and NMT models are optimized jointly and boost each other incrementally in a unified EM framework. In this way, (1) the negative effect caused by errors in the iterative back-translation process can be alleviated timely by SMT filtering noises from its phrase tables; meanwhile, (2) NMT can compensate for the deficiency of fluency inherent in SMT. Experiments conducted on en-fr and en-de translation tasks show that our method outperforms the strong baseline and achieves new state-of-the-art unsupervised machine translation performance.

scholarly journals Building Leafs Nation: rhetorical devices in the online imagined community of the Toronto Maple Leafs

2021 ◽  
Author(s):  
Nicole Iantorno
Keyword(s):  
Textual Analysis ◽  
Imagined Community ◽  
Rhetorical Devices ◽  
Communication Practices ◽  
The Creation ◽  
Words And Images ◽  
Sports Organization ◽  
Rhetorical Tropes ◽  
E Mail ◽  
The Way

Established almost 100 years ago, the Toronto Maple Leafs hockey team is known for their large, dedicated fan base called Leafs Nation. I am a devoted member of this nation and, as such, the team’s communication practices speak both to me, and about me. This major research paper (MRP) is an analysis of e-mails sent to a subscriber-only list in the context of marketing. Researching the e-mail communication from the Toronto Maple Leafs to their fan base lends itself to an understanding of the communication that occurs in a professional setting. Not only do the Toronto Maple Leafs communicate directly with their proactive fan base but I argue that the way in which they do this instills a sense of community within Leafs Nation through the use of themes, metaphors and rhetorical tropes. Communicating effectively with a fan base is an essential component in running a sports organization. Texts in the form of words and images do not only assist in getting an organization’s message to the supporters, but their connotative meanings can also contribute to the senses of community and belonging. This paper will examine how the Toronto Maple Leafs employ rhetorical devices in the e-mail newsletters sent out to Leafs Nation, as well as analyzing the rhetorical connotations in these devices. Also, I will be examining the way in which the use of rhetorical devices contributes to the creation of an online ‘imagined community,’ a concept first introduced by Benedict Anderson in 1936 in the context of nations and nationalism. Anderson stated that an imagined community does not conform to traditional ideals of a community and is constructed by those that see themselves as being a part of this community, and I see the Leafs Nation as conforming to the ideals detailed by Anderson. As such, I will be completing a qualitative textual analysis of 43 e-mails that have gone out to the subscriber-only fan list since 2012. By examining these e-mails I will attempt to identify the presence of the rhetorical devices of pathopoeia, scesis onomaton and principle of scarcity and the overall frequency with which they appear. Based on the data that emerges from my research, I will then attempt to draw correlations between the findings and attempt to link the presence of rhetorical devices as a contributing factor to the creation of Leafs Nation as an online imagined community through a qualitative textual analysis.

scholarly journals Emerging Cross-lingual Structure in Pretrained Language Models

2020 ◽  
Author(s):  
Alexis Conneau ◽  
Shijie Wu ◽  
Haoran Li ◽  
Luke Zettlemoyer ◽  
Veselin Stoyanov
Keyword(s):  
Language Models ◽  
Cross Lingual

scholarly journals Building Leafs Nation: rhetorical devices in the online imagined community of the Toronto Maple Leafs

2021 ◽  
Author(s):  
Nicole Iantorno
Keyword(s):  
Textual Analysis ◽  
Imagined Community ◽  
Rhetorical Devices ◽  
Communication Practices ◽  
The Creation ◽  
Words And Images ◽  
Sports Organization ◽  
Rhetorical Tropes ◽  
E Mail ◽  
The Way

Established almost 100 years ago, the Toronto Maple Leafs hockey team is known for their large, dedicated fan base called Leafs Nation. I am a devoted member of this nation and, as such, the team’s communication practices speak both to me, and about me. This major research paper (MRP) is an analysis of e-mails sent to a subscriber-only list in the context of marketing. Researching the e-mail communication from the Toronto Maple Leafs to their fan base lends itself to an understanding of the communication that occurs in a professional setting. Not only do the Toronto Maple Leafs communicate directly with their proactive fan base but I argue that the way in which they do this instills a sense of community within Leafs Nation through the use of themes, metaphors and rhetorical tropes. Communicating effectively with a fan base is an essential component in running a sports organization. Texts in the form of words and images do not only assist in getting an organization’s message to the supporters, but their connotative meanings can also contribute to the senses of community and belonging. This paper will examine how the Toronto Maple Leafs employ rhetorical devices in the e-mail newsletters sent out to Leafs Nation, as well as analyzing the rhetorical connotations in these devices. Also, I will be examining the way in which the use of rhetorical devices contributes to the creation of an online ‘imagined community,’ a concept first introduced by Benedict Anderson in 1936 in the context of nations and nationalism. Anderson stated that an imagined community does not conform to traditional ideals of a community and is constructed by those that see themselves as being a part of this community, and I see the Leafs Nation as conforming to the ideals detailed by Anderson. As such, I will be completing a qualitative textual analysis of 43 e-mails that have gone out to the subscriber-only fan list since 2012. By examining these e-mails I will attempt to identify the presence of the rhetorical devices of pathopoeia, scesis onomaton and principle of scarcity and the overall frequency with which they appear. Based on the data that emerges from my research, I will then attempt to draw correlations between the findings and attempt to link the presence of rhetorical devices as a contributing factor to the creation of Leafs Nation as an online imagined community through a qualitative textual analysis.

scholarly journals The Impact of Pretrained Language Models on Negation and Speculation Detection in Cross-Lingual Medical Text: Comparative Study

10.2196/18953 ◽  
2020 ◽  
Vol 8 (12) ◽  
pp. e18953
Author(s):  
Renzo Rivera Zavala ◽  
Paloma Martinez
Keyword(s):  
Machine Learning ◽  
Information Extraction ◽  
Fine Tuning ◽  
Entity Recognition ◽  
Language Models ◽  
Special Focus ◽  
Rule Based ◽  
Clinical Narrative ◽  
Cross Lingual ◽  
The Impact

Background Negation and speculation are critical elements in natural language processing (NLP)-related tasks, such as information extraction, as these phenomena change the truth value of a proposition. In the clinical narrative that is informal, these linguistic facts are used extensively with the objective of indicating hypotheses, impressions, or negative findings. Previous state-of-the-art approaches addressed negation and speculation detection tasks using rule-based methods, but in the last few years, models based on machine learning and deep learning exploiting morphological, syntactic, and semantic features represented as spare and dense vectors have emerged. However, although such methods of named entity recognition (NER) employ a broad set of features, they are limited to existing pretrained models for a specific domain or language. Objective As a fundamental subsystem of any information extraction pipeline, a system for cross-lingual and domain-independent negation and speculation detection was introduced with special focus on the biomedical scientific literature and clinical narrative. In this work, detection of negation and speculation was considered as a sequence-labeling task where cues and the scopes of both phenomena are recognized as a sequence of nested labels recognized in a single step. Methods We proposed the following two approaches for negation and speculation detection: (1) bidirectional long short-term memory (Bi-LSTM) and conditional random field using character, word, and sense embeddings to deal with the extraction of semantic, syntactic, and contextual patterns and (2) bidirectional encoder representations for transformers (BERT) with fine tuning for NER. Results The approach was evaluated for English and Spanish languages on biomedical and review text, particularly with the BioScope corpus, IULA corpus, and SFU Spanish Review corpus, with F-measures of 86.6%, 85.0%, and 88.1%, respectively, for NeuroNER and 86.4%, 80.8%, and 91.7%, respectively, for BERT. Conclusions These results show that these architectures perform considerably better than the previous rule-based and conventional machine learning–based systems. Moreover, our analysis results show that pretrained word embedding and particularly contextualized embedding for biomedical corpora help to understand complexities inherent to biomedical text.

Dual Gated Graph Attention Networks with Dynamic Iterative Training for Cross-Lingual Entity Alignment

2022 ◽  
Vol 40 (3) ◽  
pp. 1-30
Author(s):  
Zhiwen Xie ◽  
Runjie Zhu ◽  
Kunsong Zhao ◽  
Jin Liu ◽  
Guangyou Zhou ◽  
...  
Keyword(s):  
Neural Network ◽  
Structural Information ◽  
Contextual Information ◽  
Network Models ◽  
Neural Network Models ◽  
Attention Networks ◽  
Individual Knowledge ◽  
Benchmark Datasets ◽  
Cross Lingual ◽  
Network Approaches

Cross-lingual entity alignment has attracted considerable attention in recent years. Past studies using conventional approaches to match entities share the common problem of missing important structural information beyond entities in the modeling process. This allows graph neural network models to step in. Most existing graph neural network approaches model individual knowledge graphs (KGs) separately with a small amount of pre-aligned entities served as anchors to connect different KG embedding spaces. However, this characteristic can cause several major problems, including performance restraint due to the insufficiency of available seed alignments and ignorance of pre-aligned links that are useful in contextual information in-between nodes. In this article, we propose DuGa-DIT, a dual gated graph attention network with dynamic iterative training, to address these problems in a unified model. The DuGa-DIT model captures neighborhood and cross-KG alignment features by using intra-KG attention and cross-KG attention layers. With the dynamic iterative process, we can dynamically update the cross-KG attention score matrices, which enables our model to capture more cross-KG information. We conduct extensive experiments on two benchmark datasets and a case study in cross-lingual personalized search. Our experimental results demonstrate that DuGa-DIT outperforms state-of-the-art methods.

Principles of Creating an Electronic Educational Space in Humanities

2019 ◽  
Vol 8 (1) ◽  
pp. 4-10
Author(s):  
Е. Новикова ◽  
E. Novikova
Keyword(s):  
Learning Process ◽  
Structural Information ◽  
Synergetic Effect ◽  
Approaches To Learning ◽  
Systematic Application ◽  
Educational Trajectories ◽  
The Creation ◽  
The University ◽  
University Modernization

The article deals with the approaches to the creation of electronic educational space of the University. Modernization of education requires the implementation of multiple approaches to learning aimed at building the professional potential of students. The main approaches are competence-based, motivational-personal, activity-based, cognitive, structural, information-technical, sociocultural. Systematic application of approaches provides a synergetic effect in the learning process. The variability of methods for creating electronic educational space is also aimed at individualization of learning in accordance with educational trajectories.

scholarly journals Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

2021 ◽  
Vol 11 (5) ◽  
pp. 1974 ◽  
Author(s):  
Chanhee Lee ◽  
Kisu Yang ◽  
Taesun Whang ◽  
Chanjun Park ◽  
Andrew Matteson ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Language Model ◽  
Language Modeling ◽  
Language Models ◽  
Low Resource ◽  
High Resource ◽  
Cross Lingual ◽  
Data Efficiency

Language model pretraining is an effective method for improving the performance of downstream natural language processing tasks. Even though language modeling is unsupervised and thus collecting data for it is relatively less expensive, it is still a challenging process for languages with limited resources. This results in great technological disparity between high- and low-resource languages for numerous downstream natural language processing tasks. In this paper, we aim to make this technology more accessible by enabling data efficient training of pretrained language models. It is achieved by formulating language modeling of low-resource languages as a domain adaptation task using transformer-based language models pretrained on corpora of high-resource languages. Our novel cross-lingual post-training approach selectively reuses parameters of the language model trained on a high-resource language and post-trains them while learning language-specific parameters in the low-resource language. We also propose implicit translation layers that can learn linguistic differences between languages at a sequence level. To evaluate our method, we post-train a RoBERTa model pretrained in English and conduct a case study for the Korean language. Quantitative results from intrinsic and extrinsic evaluations show that our method outperforms several massively multilingual and monolingual pretrained language models in most settings and improves the data efficiency by a factor of up to 32 compared to monolingual training.

Sign in / Sign up

Export Citation Format

Share Document