scholarly journals Assessing the Impact of Translation Errors on Machine Translation Quality with Mixed-effects Models

2014 ◽  
Author(s):  
Marcello Federico ◽  
Matteo Negri ◽  
Luisa Bentivogli ◽  
Marco Turchi
Keyword(s):  
Machine Translation ◽  
Mixed Effects ◽  
Mixed Effects Models ◽  
Translation Quality ◽  
Translation Errors ◽  
The Impact

scholarly journals Evaluating the Impact of Integrating Similar Translations into Neural Machine Translation

Information ◽  
2022 ◽  
Vol 13 (1) ◽  
pp. 19
Author(s):  
Arda Tezcan ◽  
Bram Bulté
Keyword(s):  
Error Analysis ◽  
Machine Translation ◽  
Data Augmentation ◽  
Training Data ◽  
Quality Improvements ◽  
Translation Quality ◽  
Automated Evaluation ◽  
Translation Errors ◽  
Different Characteristics ◽  
The Impact

Previous research has shown that simple methods of augmenting machine translation training data and input sentences with translations of similar sentences (or fuzzy matches), retrieved from a translation memory or bilingual corpus, lead to considerable improvements in translation quality, as assessed by a limited set of automatic evaluation metrics. In this study, we extend this evaluation by calculating a wider range of automated quality metrics that tap into different aspects of translation quality and by performing manual MT error analysis. Moreover, we investigate in more detail how fuzzy matches influence translations and where potential quality improvements could still be made by carrying out a series of quantitative analyses that focus on different characteristics of the retrieved fuzzy matches. The automated evaluation shows that the quality of NFR translations is higher than the NMT baseline in terms of all metrics. However, the manual error analysis did not reveal a difference between the two systems in terms of total number of translation errors; yet, different profiles emerged when considering the types of errors made. Finally, in our analysis of how fuzzy matches influence NFR translations, we identified a number of features that could be used to improve the selection of fuzzy matches for NFR data augmentation.

scholarly journals Association of body condition with lameness in dairy cattle: a single-farm longitudinal study

2021 ◽  
pp. 1-4
Author(s):  
Michaela Kranepuhl ◽  
Detlef May ◽  
Edna Hillmann ◽  
Lorenz Gygax
Keyword(s):  
Body Condition ◽  
Body Condition Score ◽  
Mixed Effects ◽  
Mixed Effects Models ◽  
Linear Mixed Effects Models ◽  
Linear Mixed Effects ◽  
Confounding Variables ◽  
Condition Score ◽  
The Impact ◽  
The Relationship

Abstract This research communication describes the relationship between the occurrence of lameness and body condition score (BCS) in a sample of 288 cows from a single farm that were repeatedly scored in the course of 9 months while controlling for confounding variables. The relationship between BCS and lameness was evaluated using generalised linear mixed-effects models. It was found that the proportion of lame cows was higher with decreasing but also with increasing BCS, increased with lactation number and decreased with time since the last claw trimming. This is likely to reflect the importance of sufficient body condition in the prevention of lameness but also raises the question of the impact of overcondition on lameness and the influence of claw trimming events on the assessment of lameness. A stronger focus on BCS might allow improved management of lameness that is still one of the major problems in housed cows.

scholarly journals Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation

2020 ◽  
Vol 34 (01) ◽  
pp. 198-205
Author(s):  
Chenze Shao ◽  
Jinchao Zhang ◽  
Yang Feng ◽  
Fandong Meng ◽  
Jie Zhou
Keyword(s):  
Machine Translation ◽  
Cross Entropy ◽  
Sequential Dependency ◽  
Weak Correlation ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
Word Level ◽  
Translation Errors ◽  
Training Objective ◽  
Target Side

Non-Autoregressive Neural Machine Translation (NAT) achieves significant decoding speedup through generating target words independently and simultaneously. However, in the context of non-autoregressive translation, the word-level cross-entropy loss cannot model the target-side sequential dependency properly, leading to its weak correlation with the translation quality. As a result, NAT tends to generate influent translations with over-translation and under-translation errors. In this paper, we propose to train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The bag-of-ngrams training objective is differentiable and can be efficiently calculated, which encourages NAT to capture the target-side sequential dependency and correlates well with the translation quality. We validate our approach on three translation tasks and show that our approach largely outperforms the NAT baseline by about 5.0 BLEU scores on WMT14 En↔De and about 2.5 BLEU scores on WMT16 En↔Ro.

scholarly journals Accounting for external factors and early intervention adoption in the design and analysis of stepped-wedge designs: Application to a proposed study design to reduce opioid-related mortality

2020 ◽  
Author(s):  
Lior Rennert ◽  
Moonseong Heo ◽  
Alain H Litwin ◽  
Victor de Grutolla
Keyword(s):  
Mixed Effects ◽  
External Factors ◽  
Mixed Effects Models ◽  
Confounding Factors ◽  
Intervention Effect ◽  
Type 1 Error ◽  
Inflated Type ◽  
Unbiased Estimates ◽  
The Impact

Background: Stepped-wedge designs (SWDs) are currently being used to investigate interventions to reduce opioid overdose deaths in communities located in several states. However, these interventions are competing with external factors such as newly initiated public policies limiting opioid prescriptions, media awareness campaigns, and social distancing orders due to the COVID-19 pandemic. Furthermore, control communities may prematurely adopt components of the proposed intervention as they become widely available. These types of events induce confounding of the intervention effect by time. Such confounding is a well-known limitation of SWDs; a common approach to adjusting for it makes use of a mixed effects modeling framework that includes both fixed and random effects for time. However, these models have several shortcomings when multiple confounding factors are present. Methods: We discuss the limitations of existing methods based on mixed effects models in the context of proposed SWDs to investigate interventions intended to reduce mortality associated with the opioid epidemic, and propose solutions to accommodate deviations from assumptions that underlie these models. We conduct an extensive simulation study of anticipated data from SWD trials targeting the current opioid epidemic in order to examine the performance of these models under different sources of confounding. We specifically examine the impact of factors external to the study and premature adoption of intervention components. Results: When only external factors are present, our simulation studies show that commonly used mixed effects models can result in unbiased estimates of the intervention effect, but have inflated Type 1 error and result in under coverage of confidence intervals. These models are severely biased when confounding factors differentially impact intervention and control clusters; premature adoption of intervention components is an example of this scenario. In these scenarios, models that incorporate fixed intervention-by-time interaction terms and an unstructured covariance for the intervention-by-cluster-by-time random effects result in unbiased estimates of the intervention effect, reach nominal confidence interval coverage, and preserve Type 1 error, but may reduce power. Conclusions: The incorporation of fixed and random time effects in mixed effects models require certain assumptions about the impact of confounding by time in SWD. Violations of these assumptions can result in severe bias of the intervention effect estimate, under coverage of confidence intervals, and inflated Type 1 error. Since model choice has considerable impact on study power as well as validity of results, careful consideration needs to be given to choosing an appropriate model that takes into account potential confounding factors.

scholarly journals Towards a Better Integration of Fuzzy Matches in Neural Machine Translation through Data Augmentation

Informatics ◽  
2021 ◽  
Vol 8 (1) ◽  
pp. 7
Author(s):  
Arda Tezcan ◽  
Bram Bulté ◽  
Bram Vanroy
Keyword(s):  
Machine Translation ◽  
Data Augmentation ◽  
Sentence Length ◽  
Added Value ◽  
Neural Machine Translation ◽  
Combination Technique ◽  
Translation Quality ◽  
Fuzzy Match ◽  
The Impact ◽  
Matching Techniques

We identify a number of aspects that can boost the performance of Neural Fuzzy Repair (NFR), an easy-to-implement method to integrate translation memory matches and neural machine translation (NMT). We explore various ways of maximising the added value of retrieved matches within the NFR paradigm for eight language combinations, using Transformer NMT systems. In particular, we test the impact of different fuzzy matching techniques, sub-word-level segmentation methods and alignment-based features on overall translation quality. Furthermore, we propose a fuzzy match combination technique that aims to maximise the coverage of source words. This is supplemented with an analysis of how translation quality is affected by input sentence length and fuzzy match score. The results show that applying a combination of the tested modifications leads to a significant increase in estimated translation quality over all baselines for all language combinations.

scholarly journals Pre-Reordering for Neural Machine Translation: Helpful or Harmful?

2017 ◽  
Vol 108 (1) ◽  
pp. 171-182 ◽  
Author(s):  
Jinhua Du ◽  
Andy Way
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Word Class ◽  
Word Embeddings ◽  
Neural Machine Translation ◽  
Parts Of Speech ◽  
Translation Quality ◽  
The Impact ◽  
Japanese English ◽  
Target Side

AbstractPre-reordering, a preprocessing to make the source-side word orders close to those of the target side, has been proven very helpful for statistical machine translation (SMT) in improving translation quality. However, is it the case in neural machine translation (NMT)? In this paper, we firstly investigate the impact of pre-reordered source-side data on NMT, and then propose to incorporate features for the pre-reordering model in SMT as input factors into NMT (factored NMT). The features, namely parts-of-speech (POS), word class and reordered index, are encoded as feature vectors and concatenated to the word embeddings to provide extra knowledge for NMT. Pre-reordering experiments conducted on Japanese↔English and Chinese↔English show that pre-reordering the source-side data for NMT is redundant and NMT models trained on pre-reordered data deteriorate translation performance. However, factored NMT using SMT-based pre-reordering features on Japanese→English and Chinese→English is beneficial and can further improve by 4.48 and 5.89 relative BLEU points, respectively, compared to the baseline NMT system.

scholarly journals A Systematic Study of Inner-Attention-Based Sentence Representations in Multilingual Neural Machine Translation

2020 ◽  
Vol 46 (2) ◽  
pp. 387-424 ◽  
Author(s):  
Raúl Vázquez ◽  
Alessandro Raganato ◽  
Mathias Creutz ◽  
Jörg Tiedemann
Keyword(s):  
Machine Translation ◽  
Improve Performance ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
Intermediate Layers ◽  
Depth Analysis ◽  
Classification Tasks ◽  
The Impact ◽  
Meaning Representation

Neural machine translation has considerably improved the quality of automatic translations by learning good representations of input sentences. In this article, we explore a multilingual translation model capable of producing fixed-size sentence representations by incorporating an intermediate crosslingual shared layer, which we refer to as attention bridge. This layer exploits the semantics from each language and develops into a language-agnostic meaning representation that can be efficiently used for transfer learning. We systematically study the impact of the size of the attention bridge and the effect of including additional languages in the model. In contrast to related previous work, we demonstrate that there is no conflict between translation performance and the use of sentence representations in downstream tasks. In particular, we show that larger intermediate layers not only improve translation quality, especially for long sentences, but also push the accuracy of trainable classification tasks. Nevertheless, shorter representations lead to increased compression that is beneficial in non-trainable similarity tasks. Similarly, we show that trainable downstream tasks benefit from multilingual models, whereas additional language signals do not improve performance in non-trainable benchmarks. This is an important insight that helps to properly design models for specific applications. Finally, we also include an in-depth analysis of the proposed attention bridge and its ability to encode linguistic properties. We carefully analyze the information that is captured by individual attention heads and identify interesting patterns that explain the performance of specific settings in linguistic probing tasks.

scholarly journals “Bilingual Expert” Can Find Translation Errors

2019 ◽  
Vol 33 ◽  
pp. 6367-6374 ◽  
Author(s):  
Kai Fan ◽  
Jiayi Wang ◽  
Bo Li ◽  
Fengming Zhou ◽  
Boxing Chen ◽  
...  
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Language Model ◽  
Target Language ◽  
Quality Estimation ◽  
Parallel Corpora ◽  
Translation Quality ◽  
Expert Model ◽  
Parallel Data ◽  
Translation Errors

The performances of machine translation (MT) systems are usually evaluated by the metric BLEU when the golden references are provided. However, in the case of model inference or production deployment, golden references are usually expensively available, such as human annotation with bilingual expertise. In order to address the issue of translation quality estimation (QE) without reference, we propose a general framework for automatic evaluation of the translation output for the QE task in the Conference on Statistical Machine Translation (WMT). We first build a conditional target language model with a novel bidirectional transformer, named neural bilingual expert model, which is pre-trained on large parallel corpora for feature extraction. For QE inference, the bilingual expert model can simultaneously produce the joint latent representation between the source and the translation, and real-valued measurements of possible erroneous tokens based on the prior knowledge learned from parallel data. Subsequently, the features will further be fed into a simple Bi-LSTM predictive model for quality estimation. The experimental results show that our approach achieves the state-of-the-art performance in most public available datasets of WMT 2017/2018 QE task.

Sign in / Sign up

Export Citation Format

Share Document