Neural Machine Translation Using Attention Mechanism

2022 ◽  
pp. 717-729
Author(s):  
Sai Yashwanth Velpuri ◽  
Sonakshi Karanwal ◽  
R. Anita
Keyword(s):  
Machine Translation ◽  
Attention Mechanism ◽  
Neural Machine Translation

Deep Learning-based Roman-Urdu to Urdu Transliteration

2020 ◽  
pp. 2152001
Author(s):  
Mehreen Alam ◽  
Sibt ul Hussain
Keyword(s):  
Machine Translation ◽  
State Of The Art ◽  
Research Problem ◽  
Attention Mechanism ◽  
Data Driven ◽  
Neural Machine Translation ◽  
Parallel Corpus ◽  
Source Language ◽  
Data Driven Approach ◽  
Modern Machine

Attention-based encoder-decoder models have superseded conventional techniques due to their unmatched performance on many neural machine translation problems. Usually, the encoders and decoders are two recurrent neural networks where the decoder is directed to focus on relevant parts of the source language using attention mechanism. This data-driven approach leads to generic and scalable solutions with no reliance on manual hand-crafted features. To the best of our knowledge, none of the modern machine translation approaches has been applied to address the research problem of Urdu machine transliteration. Ours is the first attempt to apply the deep neural network-based encoder-decoder using attention mechanism to address the aforementioned problem using Roman-Urdu and Urdu parallel corpus. To this end, we present (i) the first ever Roman-Urdu to Urdu parallel corpus of 1.1 million sentences, (ii) three state of the art encoder-decoder models, and (iii) a detailed empirical analysis of these three models on the Roman-Urdu to Urdu parallel corpus. Overall, attention-based model gives state-of-the-art performance with the benchmark of 70 BLEU score. Our qualitative experimental evaluation shows that our models generate coherent transliterations which are grammatically and logically correct.

scholarly journals Analyses and Modeling of Neural Machine Translation for English-to-Khasi

2020 ◽  
Vol 9 (2) ◽  
pp. 115-118
Keyword(s):  
Deep Learning ◽  
Machine Translation ◽  
Statistical Machine Translation ◽  
Language Barrier ◽  
Attention Mechanism ◽  
Training Dataset ◽  
Learning Approach ◽  
Neural Machine Translation ◽  
Community Or ◽  
North East

Language barrier is a common issue faced by humans who move from one community or group to another. Statistical machine translation has enabled us to solve this issue to a certain extent, by formulating models to translate text from one language to another. Statistical machine translation has come a long way but they have their limitations in terms of translating words that belongs to an entirely different context that is not available in the training dataset. This has paved way for neural Machine Translation (NMT), a deep learning approach in solving sequence to sequence translation. Khasi is a language popularly spoken in Meghalaya, a north-east state in India. Its wide and unexplored. In this paper we will discuss about the modeling and analyzing of a NMT base model and a NMT model using Attention mechanism for English to Khasi.

scholarly journals Guiding Attention in Sequence-to-Sequence Models for Dialogue Act Prediction

2020 ◽  
Vol 34 (05) ◽  
pp. 7594-7601
Author(s):  
Pierre Colombo ◽  
Emile Chapuis ◽  
Matteo Manica ◽  
Emmanuel Vignon ◽  
Giovanna Varni ◽  
...  
Keyword(s):  
Machine Translation ◽  
Random Fields ◽  
Conditional Random Fields ◽  
State Of The Art ◽  
The State ◽  
Attention Mechanism ◽  
Accuracy Score ◽  
Beam Search ◽  
Conversational Agents ◽  
Neural Machine Translation

The task of predicting dialog acts (DA) based on conversational dialog is a key component in the development of conversational agents. Accurately predicting DAs requires a precise modeling of both the conversation and the global tag dependencies. We leverage seq2seq approaches widely adopted in Neural Machine Translation (NMT) to improve the modelling of tag sequentiality. Seq2seq models are known to learn complex global dependencies while currently proposed approaches using linear conditional random fields (CRF) only model local tag dependencies. In this work, we introduce a seq2seq model tailored for DA classification using: a hierarchical encoder, a novel guided attention mechanism and beam search applied to both training and inference. Compared to the state of the art our model does not require handcrafted features and is trained end-to-end. Furthermore, the proposed approach achieves an unmatched accuracy score of 85% on SwDA, and state-of-the-art accuracy score of 91.6% on MRDA.

PhraseAttn: Dynamic Slot Capsule Networks for phrase representation in Neural Machine Translation

2021 ◽  
pp. 1-8
Author(s):  
Binh Nguyen ◽  
Binh Le ◽  
Long H.B. Nguyen ◽  
Dien Dinh
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Machine Translation ◽  
Language Processing ◽  
Vital Role ◽  
Attention Mechanism ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Word Representation

 Word representation plays a vital role in most Natural Language Processing systems, especially for Neural Machine Translation. It tends to capture semantic and similarity between individual words well, but struggle to represent the meaning of phrases or multi-word expressions. In this paper, we investigate a method to generate and use phrase information in a translation model. To generate phrase representations, a Primary Phrase Capsule network is first employed, then iteratively enhancing with a Slot Attention mechanism. Experiments on the IWSLT English to Vietnamese, French, and German datasets show that our proposed method consistently outperforms the baseline Transformer, and attains competitive results over the scaled Transformer with two times lower parameters.

A comparative study of neural machine translation models for Turkish language

2021 ◽  
pp. 1-11
Author(s):  
Özgür Özdemir ◽  
Emre Salih Akın ◽  
Rıza Velioğlu ◽  
Tuğba Dalyan
Keyword(s):  
Machine Translation ◽  
Computational Linguistics ◽  
Attention Mechanism ◽  
Neural Machine Translation ◽  
English Translations ◽  
Benchmark Datasets ◽  
Important Challenge ◽  
Morphologically Rich Languages ◽  
Transformer Model ◽  
Ted Talks

Machine translation (MT) is an important challenge in the fields of Computational Linguistics. In this study, we conducted neural machine translation (NMT) experiments on two different architectures. First, Sequence to Sequence (Seq2Seq) architecture along with a variation that utilizes attention mechanism is performed on translation task. Second, an architecture that is fully based on the self-attention mechanism, namely Transformer, is employed to perform a comprehensive comparison. Besides, the contribution of employing Byte Pair Encoding (BPE) and Gumbel Softmax distributions are examined for both architectures. The experiments are conducted on two different datasets: TED Talks that is one of the popular benchmark datasets for NMT especially among morphologically rich languages like Turkish and WMT18 News dataset that is provided by The Third Conference on Machine Translation (WMT) for shared tasks on various aspects of machine translation. The evaluation of Turkish-to-English translations’ results demonstrate that the Transformer model with combination of BPE and Gumbel Softmax achieved 22.4 BLEU score on TED Talks and 38.7 BLUE score on WMT18 News dataset. The empirical results support that using Gumbel Softmax distribution improves the quality of translations for both architectures.

scholarly journals Dual contextual module for neural machine translation

2021 ◽  
Author(s):  
Isaac Kojo Essel Ampomah ◽  
Sally McClean ◽  
Glenn Hawe
Keyword(s):  
Machine Translation ◽  
Contextual Information ◽  
Attention Mechanism ◽  
The Self ◽  
Experimental Results ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Global Context ◽  
Overall Performance ◽  
Transformer Model

AbstractSelf-attention-based encoder-decoder frameworks have drawn increasing attention in recent years. The self-attention mechanism generates contextual representations by attending to all tokens in the sentence. Despite improvements in performance, recent research argues that the self-attention mechanism tends to concentrate more on the global context with less emphasis on the contextual information available within the local neighbourhood of tokens. This work presents the Dual Contextual (DC) module, an extension of the conventional self-attention unit, to effectively leverage both the local and global contextual information. The goal is to further improve the sentence representation ability of the encoder and decoder subnetworks, thus enhancing the overall performance of the translation model. Experimental results on WMT’14 English-German (En$$\rightarrow $$ → De) and eight IWSLT translation tasks show that the DC module can further improve the translation performance of the Transformer model.

scholarly journals Generating Alignments Using Target Foresight in Attention-Based Neural Machine Translation

2017 ◽  
Vol 108 (1) ◽  
pp. 27-36
Author(s):  
Jan-Thorsten Peter ◽  
Arne Nix ◽  
Hermann Ney
Keyword(s):  
Target Word ◽  
Machine Translation ◽  
Explicit Knowledge ◽  
Attention Mechanism ◽  
Target Information ◽  
High Quality ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
Successful Approach

AbstractNeural machine translation (NMT) has shown large improvements in recent years. The currently most successful approach in this area relies on the attention mechanism, which is often interpreted as an alignment, even though it is computed without explicit knowledge of the target word. This limitation is the most likely reason that the quality of attention-based alignments is inferior to the quality of traditional alignment methods. Guided alignment training has shown that alignments are still capable of improving translation quality. In this work, we propose an extension of the attention-based NMT model that introduces target information into the attention mechanism to produce high-quality alignments. In comparison to the conventional attention-based alignments, our model halves the Aer with an absolute improvement of 19.1% Aer. Compared to GIZA++ it shows an absolute improvement of 2.0% Aer.

Sign in / Sign up

Export Citation Format

Share Document