Statistical Analysis of Machine Translation Evaluation Systems for English- Hindi Language Pair

2020 ◽  
Vol 13 (5) ◽  
pp. 864-870
Author(s):  
Pooja Malik ◽  
Y. Mrudula ◽  
Anurag S. Baghel
Keyword(s):  
Statistical Analysis ◽  
Machine Translation ◽  
Automatic Machine ◽  
Evaluation Metrics ◽  
Machine Translation Evaluation ◽  
Evaluation Systems ◽  
Evaluation Scores ◽  
Sentence Level ◽  
Set Up ◽  
Language Pair

Background: Automatic Machine Translation (AMT) Evaluation Metrics have become popular in the Machine Translation Community in recent times. This is because of the popularity of Machine Translation engines and Machine Translation as a field itself. Translator is a very important tool to break barriers between communities especially in countries like India, where people speak 22 different languages and their many variations. With the onset of Machine Translation engines, there is a need for a system that evaluates how well these are performing. This is where machine translation evaluation enters. Objective: This paper discusses the importance of Automatic Machine Translation Evaluation and compares various Machine Translation Evaluation metrics by performing Statistical Analysis on various metrics and human evaluations to find out which metric has the highest correlation with human scores. Methods: The correlation between the Automatic and Human Evaluation Scores and the correlation between the five Automatic evaluation scores are examined at the sentence level. Moreover, a hypothesis is set up and p-values are calculated to find out how significant these correlations are. Results: The results of the statistical analysis of the scores of various metrics and human scores are shown in the form of graphs to see the trend of the correlation between the scores of Automatic Machine Translation Evaluation metrics and human scores. Conclusion: Out of the five metrics considered for the study, METEOR shows the highest correlation with human scores as compared to the other metrics.

scholarly journals Discourse Structure in Machine Translation Evaluation

2017 ◽  
Vol 43 (4) ◽  
pp. 683-722 ◽  
Author(s):  
Shafiq Joty ◽  
Francisco Guzmán ◽  
Lluís Màrquez ◽  
Preslav Nakov
Keyword(s):  
Machine Translation ◽  
Similarity Measures ◽  
Discourse Structure ◽  
System Level ◽  
Structure Theory ◽  
Evaluation Metrics ◽  
Machine Translation Evaluation ◽  
Sentence Level ◽  
Relation Type ◽  
Parse Trees

In this article, we explore the potential of using sentence-level discourse structure for machine translation evaluation. We first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST). Then, we show that a simple linear combination with these measures can help improve various existing machine translation evaluation metrics regarding correlation with human judgments both at the segment level and at the system level. This suggests that discourse information is complementary to the information used by many of the existing evaluation metrics, and thus it could be taken into account when developing richer evaluation metrics, such as the WMT-14 winning combined metric DiscoTK party. We also provide a detailed analysis of the relevance of various discourse elements and relations from the RST parse trees for machine translation evaluation. In particular, we show that (i) all aspects of the RST tree are relevant, (ii) nuclearity is more useful than relation type, and (iii) the similarity of the translation RST tree to the reference RST tree is positively correlated with translation quality.

scholarly journals Using Word Embeddings to Enforce Document-Level Lexical Consistency in Machine Translation

2017 ◽  
Vol 108 (1) ◽  
pp. 85-96 ◽  
Author(s):  
Eva Martínez Garcia ◽  
Carles Creus ◽  
Cristina España-Bonet ◽  
Lluís Màrquez
Keyword(s):  
Machine Translation ◽  
Evaluation Metrics ◽  
Automatic Evaluation ◽  
Word Embeddings ◽  
Standard Document ◽  
Sentence Level ◽  
Word Translation ◽  
Stochastic Mechanism ◽  
Document Level

Abstract We integrate new mechanisms in a document-level machine translation decoder to improve the lexical consistency of document translations. First, we develop a document-level feature designed to score the lexical consistency of a translation. This feature, which applies to words that have been translated into different forms within the document, uses word embeddings to measure the adequacy of each word translation given its context. Second, we extend the decoder with a new stochastic mechanism that, at translation time, allows to introduce changes in the translation oriented to improve its lexical consistency. We evaluate our system on English–Spanish document translation, and we conduct automatic and manual assessments of its quality. The automatic evaluation metrics, applied mainly at sentence level, do not reflect significant variations. On the contrary, the manual evaluation shows that the system dealing with lexical consistency is preferred over both a standard sentence-level and a standard document-level phrase-based MT systems.

scholarly journals COSTA MT Evaluation Tool: An Open Toolkit for Human Machine Translation Evaluation

2013 ◽  
Vol 100 (1) ◽  
pp. 83-89 ◽  
Author(s):  
Konstantinos Chatzitheodorou
Keyword(s):  
Machine Translation ◽  
Evaluation Tool ◽  
Machine Translation Evaluation ◽  
Translation Error ◽  
Mt Evaluation ◽  
Java Program ◽  
Error Classification ◽  
The One ◽  
Language Pair

Abstract A hotly debated topic in machine translation is human evaluation. On the one hand, it is extremely costly and time consuming; on the other, it is an important and unfortunately inevitable part of any system. This paper describes COSTA MT Evaluation Tool, an open stand-alone tool for human machine translation evaluation. It is a Java program that can be used to manually evaluate the quality of the machine translation output. It is simple in use, designed to allow machine translation potential users and developers to analyze their systems using a friendly environment. It enables the ranking of the quality of machine translation output segment-bysegment for a particular language pair. The benefits of this tool are multiple. Firstly, it is a rich repository of commonly used industry criteria (fluency, adequacy and translation error classification). Secondly, it is freely available to anyone and provides results that can be further analyzed. Thirdly, it estimates the time needed for each evaluated sentence. Finally, it gives suggestions about the fuzzy matching of the candidate translations.

Sign in / Sign up

Export Citation Format

Share Document