scholarly journals Dual contextual module for neural machine translation

2021 ◽  
Author(s):  
Isaac Kojo Essel Ampomah ◽  
Sally McClean ◽  
Glenn Hawe
Keyword(s):  
Machine Translation ◽  
Contextual Information ◽  
Attention Mechanism ◽  
The Self ◽  
Experimental Results ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Global Context ◽  
Overall Performance ◽  
Transformer Model

AbstractSelf-attention-based encoder-decoder frameworks have drawn increasing attention in recent years. The self-attention mechanism generates contextual representations by attending to all tokens in the sentence. Despite improvements in performance, recent research argues that the self-attention mechanism tends to concentrate more on the global context with less emphasis on the contextual information available within the local neighbourhood of tokens. This work presents the Dual Contextual (DC) module, an extension of the conventional self-attention unit, to effectively leverage both the local and global contextual information. The goal is to further improve the sentence representation ability of the encoder and decoder subnetworks, thus enhancing the overall performance of the translation model. Experimental results on WMT’14 English-German (En$$\rightarrow $$ → De) and eight IWSLT translation tasks show that the DC module can further improve the translation performance of the Transformer model.

PhraseAttn: Dynamic Slot Capsule Networks for phrase representation in Neural Machine Translation

2021 ◽  
pp. 1-8
Author(s):  
Binh Nguyen ◽  
Binh Le ◽  
Long H.B. Nguyen ◽  
Dien Dinh
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Machine Translation ◽  
Language Processing ◽  
Vital Role ◽  
Attention Mechanism ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Word Representation

 Word representation plays a vital role in most Natural Language Processing systems, especially for Neural Machine Translation. It tends to capture semantic and similarity between individual words well, but struggle to represent the meaning of phrases or multi-word expressions. In this paper, we investigate a method to generate and use phrase information in a translation model. To generate phrase representations, a Primary Phrase Capsule network is first employed, then iteratively enhancing with a Slot Attention mechanism. Experiments on the IWSLT English to Vietnamese, French, and German datasets show that our proposed method consistently outperforms the baseline Transformer, and attains competitive results over the scaled Transformer with two times lower parameters.

A comparative study of neural machine translation models for Turkish language

2021 ◽  
pp. 1-11
Author(s):  
Özgür Özdemir ◽  
Emre Salih Akın ◽  
Rıza Velioğlu ◽  
Tuğba Dalyan
Keyword(s):  
Machine Translation ◽  
Computational Linguistics ◽  
Attention Mechanism ◽  
Neural Machine Translation ◽  
English Translations ◽  
Benchmark Datasets ◽  
Important Challenge ◽  
Morphologically Rich Languages ◽  
Transformer Model ◽  
Ted Talks

Machine translation (MT) is an important challenge in the fields of Computational Linguistics. In this study, we conducted neural machine translation (NMT) experiments on two different architectures. First, Sequence to Sequence (Seq2Seq) architecture along with a variation that utilizes attention mechanism is performed on translation task. Second, an architecture that is fully based on the self-attention mechanism, namely Transformer, is employed to perform a comprehensive comparison. Besides, the contribution of employing Byte Pair Encoding (BPE) and Gumbel Softmax distributions are examined for both architectures. The experiments are conducted on two different datasets: TED Talks that is one of the popular benchmark datasets for NMT especially among morphologically rich languages like Turkish and WMT18 News dataset that is provided by The Third Conference on Machine Translation (WMT) for shared tasks on various aspects of machine translation. The evaluation of Turkish-to-English translations’ results demonstrate that the Transformer model with combination of BPE and Gumbel Softmax achieved 22.4 BLEU score on TED Talks and 38.7 BLUE score on WMT18 News dataset. The empirical results support that using Gumbel Softmax distribution improves the quality of translations for both architectures.

scholarly journals English Machine Translation Model Based on an Improved Self-Attention Technology

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Wenxia Pan
Keyword(s):  
Machine Translation ◽  
Language Processing ◽  
Extraction Methods ◽  
Attention Mechanism ◽  
The Self ◽  
Position Information ◽  
Directional Information ◽  
Translation Model ◽  
Model Based ◽  
Phrase Extraction

English machine translation is a natural language processing research direction that has important scientific research value and practical value in the current artificial intelligence boom. The variability of language, the limited ability to express semantic information, and the lack of parallel corpus resources all limit the usefulness and popularity of English machine translation in practical applications. The self-attention mechanism has received a lot of attention in English machine translation tasks because of its highly parallelizable computing ability, which reduces the model’s training time and allows it to capture the semantic relevance of all words in the context. The efficiency of the self-attention mechanism, however, differs from that of recurrent neural networks because it ignores the position and structure information between context words. The English machine translation model based on the self-attention mechanism uses sine and cosine position coding to represent the absolute position information of words in order to enable the model to use position information between words. This method, on the other hand, can reflect relative distance but does not provide directionality. As a result, a new model of English machine translation is proposed, which is based on the logarithmic position representation method and the self-attention mechanism. This model retains the distance and directional information between words, as well as the efficiency of the self-attention mechanism. Experiments show that the nonstrict phrase extraction method can effectively extract phrase translation pairs from the n-best word alignment results and that the extraction constraint strategy can improve translation quality even further. Nonstrict phrase extraction methods and n-best alignment results can significantly improve the quality of translation translations when compared to traditional phrase extraction methods based on single alignment.

scholarly journals Source-Word Decomposition for Neural Machine Translation

2020 ◽  
Vol 2020 ◽  
pp. 1-10
Author(s):  
Thien Nguyen ◽  
Hoai Le ◽  
Van-Huy Pham
Keyword(s):  
Machine Translation ◽  
Linguistic Features ◽  
Specialized Knowledge ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Part Of Speech ◽  
Effective System ◽  
Feature Based ◽  
Transformer Model ◽  
Translation Systems

End-to-end neural machine translation does not require us to have specialized knowledge of investigated language pairs in building an effective system. On the other hand, feature engineering proves to be vital in other artificial intelligence fields, such as speech recognition and computer vision. Inspired by works in those fields, in this paper, we propose a novel feature-based translation model by modifying the state-of-the-art transformer model. Specifically, the encoder of the modified transformer model takes input combinations of linguistic features comprising of lemma, dependency label, part-of-speech tag, and morphological label instead of source words. The experiment results for the Russian-Vietnamese language pair show that the proposed feature-based transformer model improves over the strongest baseline transformer translation model by impressive 4.83 BLEU. In addition, experiment analysis reveals that human judgment on the translation results strongly confirms machine judgment. Our model could be useful in building translation systems translating from a highly inflectional language into a noninflectional language.

scholarly journals BioNMT: A Biomedical Neural Machine Translation System

2020 ◽  
Vol 15 (6) ◽  
Author(s):  
Hongtao Liu ◽  
Yanchun Liang ◽  
Liupu Wang ◽  
Xiaoyue Feng ◽  
Renchu Guan
Keyword(s):  
Foreign Language ◽  
Machine Translation ◽  
Deep Neural Networks ◽  
Translation System ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Machine Translation System ◽  
Biomedical Field ◽  
Biomedical Texts ◽  
Transformer Model

To solve the problem of translation of professional vocabulary in the biomedical field and help biological researchers to translate and understand foreign language documents, we proposed a semantic disambiguation model and external dictionaries to build a novel translation model for biomedical texts based on the transformer model. The proposed biomedical neural machine translation system (BioNMT) adopts the sequence-to-sequence translation framework, which is based on deep neural networks. To construct the specialized vocabulary of biology and medicine, a hybrid corpus was obtained using a crawler system extracting from universal corpus and biomedical corpus. The experimental results showed that BioNMT which composed by professional biological dictionary and Transformer model increased the bilingual evaluation understudy (BLEU) value by 14.14%, and the perplexity was reduced by 40%. And compared with Google Translation System and Baidu Translation System, BioNMT achieved better translations about paragraphs and resolve the ambiguity of biomedical name entities to greatly improved.

scholarly journals A Transformer-Based Neural Machine Translation Model for Arabic Dialects That Utilizes Subword Units

Sensors ◽  
2021 ◽  
Vol 21 (19) ◽  
pp. 6509
Author(s):  
Laith H. Baniata ◽  
Isaac. K. E. Ampomah ◽  
Seyoung Park
Keyword(s):  
Machine Translation ◽  
Target Language ◽  
Modern Standard Arabic ◽  
Neural Machine Translation ◽  
Unknown Word ◽  
Translation Model ◽  
Standard Arabic ◽  
Arabic Dialects ◽  
Transformer Model ◽  
Modern Standard

Languages that allow free word order, such as Arabic dialects, are of significant difficulty for neural machine translation (NMT) because of many scarce words and the inefficiency of NMT systems to translate these words. Unknown Word (UNK) tokens represent the out-of-vocabulary words for the reason that NMT systems run with vocabulary that has fixed size. Scarce words are encoded completely as sequences of subword pieces employing the Word-Piece Model. This research paper introduces the first Transformer-based neural machine translation model for Arabic vernaculars that employs subword units. The proposed solution is based on the Transformer model that has been presented lately. The use of subword units and shared vocabulary within the Arabic dialect (the source language) and modern standard Arabic (the target language) enhances the behavior of the multi-head attention sublayers for the encoder by obtaining the overall dependencies between words of input sentence for Arabic vernacular. Experiments are carried out from Levantine Arabic vernacular (LEV) to modern standard Arabic (MSA) and Maghrebi Arabic vernacular (MAG) to MSA, Gulf–MSA, Nile–MSA, Iraqi Arabic (IRQ) to MSA translation tasks. Extensive experiments confirm that the suggested model adequately addresses the unknown word issue and boosts the quality of translation from Arabic vernaculars to Modern standard Arabic (MSA).

scholarly journals Improving Transformer-Based Neural Machine Translation with Prior Alignments

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Thien Nguyen ◽  
Lam Nguyen ◽  
Phuoc Tran ◽  
Huu Nguyen
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
The Novel ◽  
Reverse Effect ◽  
Human Judgment ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Type Selection ◽  
Transformer Model ◽  
Positive Effect

Transformer is a neural machine translation model which revolutionizes machine translation. Compared with traditional statistical machine translation models and other neural machine translation models, the recently proposed transformer model radically and fundamentally changes machine translation with its self-attention and cross-attention mechanisms. These mechanisms effectively model token alignments between source and target sentences. It has been reported that the transformer model provides accurate posterior alignments. In this work, we empirically prove the reverse effect, showing that prior alignments help transformer models produce better translations. Experiment results on Vietnamese-English news translation task show not only the positive effect of manually annotated alignments on transformer models but also the surprising outperformance of statistically constructed alignments reinforced with the flexibility of token-type selection over manual alignments in improving transformer models. Statistically constructed word-to-lemma alignments are used to train a word-to-word transformer model. The novel hybrid transformer model improves the baseline transformer model and transformer model trained with manual alignments by 2.53 and 0.79 BLEU, respectively. In addition to BLEU score, we make limited human judgment on translation results. Strong correlation between human and machine judgment confirms our findings.

scholarly journals Context-Aware Neural Machine Translation for Korean Honorific Expressions

Electronics ◽  
2021 ◽  
Vol 10 (13) ◽  
pp. 1589
Author(s):  
Yongkeun Hwang ◽  
Yanghoon Kim ◽  
Kyomin Jung
Keyword(s):  
Machine Translation ◽  
Deep Neural Networks ◽  
Contextual Information ◽  
Context Aware ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
Sentence Level ◽  
Proposed Model ◽  
The Given ◽  
The Relationship

Neural machine translation (NMT) is one of the text generation tasks which has achieved significant improvement with the rise of deep neural networks. However, language-specific problems such as handling the translation of honorifics received little attention. In this paper, we propose a context-aware NMT to promote translation improvements of Korean honorifics. By exploiting the information such as the relationship between speakers from the surrounding sentences, our proposed model effectively manages the use of honorific expressions. Specifically, we utilize a novel encoder architecture that can represent the contextual information of the given input sentences. Furthermore, a context-aware post-editing (CAPE) technique is adopted to refine a set of inconsistent sentence-level honorific translations. To demonstrate the efficacy of the proposed method, honorific-labeled test data is required. Thus, we also design a heuristic that labels Korean sentences to distinguish between honorific and non-honorific styles. Experimental results show that our proposed method outperforms sentence-level NMT baselines both in overall translation quality and honorific translations.

scholarly journals Efficient Context-Aware Neural Machine Translation with Layer-Wise Weighting and Input-Aware Gating

2020 ◽  
Author(s):  
Hongfei Xu ◽  
Deyi Xiong ◽  
Josef van Genabith ◽  
Qiuhui Liu
Keyword(s):  
Machine Translation ◽  
Contextual Information ◽  
Computational Cost ◽  
Representation Learning ◽  
Vital Role ◽  
Context Aware ◽  
Neural Machine Translation ◽  
Gating Mechanism ◽  
Sentence Level ◽  
Parallel Data

Existing Neural Machine Translation (NMT) systems are generally trained on a large amount of sentence-level parallel data, and during prediction sentences are independently translated, ignoring cross-sentence contextual information. This leads to inconsistency between translated sentences. In order to address this issue, context-aware models have been proposed. However, document-level parallel data constitutes only a small part of the parallel data available, and many approaches build context-aware models based on a pre-trained frozen sentence-level translation model in a two-step training manner. The computational cost of these approaches is usually high. In this paper, we propose to make the most of layers pre-trained on sentence-level data in contextual representation learning, reusing representations from the sentence-level Transformer and significantly reducing the cost of incorporating contexts in translation. We find that representations from shallow layers of a pre-trained sentence-level encoder play a vital role in source context encoding, and propose to perform source context encoding upon weighted combinations of pre-trained encoder layers' outputs. Instead of separately performing source context and input encoding, we propose to iteratively and jointly encode the source input and its contexts and to generate input-aware context representations with a cross-attention layer and a gating mechanism, which resets irrelevant information in context encoding. Our context-aware Transformer model outperforms the recent CADec [Voita et al., 2019c] on the English-Russian subtitle data and is about twice as fast in training and decoding.

Sign in / Sign up

Export Citation Format

Share Document