scholarly journals Evaluating MT systems with BEER

2015 ◽  
Vol 104 (1) ◽  
pp. 17-26
Author(s):  
Miloš Stanojević ◽  
Khalil Sima’an
Keyword(s):  
Open Source ◽  
Machine Translation ◽  
Word Order ◽  
Learning To Rank ◽  
Training Methods ◽  
Machine Translation Evaluation ◽  
Evaluation Metric ◽  
Hierarchical Representations

Abstract We present BEER, an open source implementation of a machine translation evaluation metric. BEER is a metric trained for high correlation with human ranking by using learning-to-rank training methods. For evaluation of lexical accuracy it uses sub-word units (character n-grams) while for measuring word order it uses hierarchical representations based on PETs (permutation trees). During the last WMT metrics tasks, BEER has shown high correlation with human judgments both on the sentence and the corpus levels. In this paper we will show how BEER can be used for (i) full evaluation of MT output, (ii) isolated evaluation of word order and (iii) tuning MT systems.

scholarly journals Z-MERT: A Fully Configurable Open Source Tool for Minimum Error Rate Training of Machine Translation Systems

2009 ◽  
Vol 91 (1) ◽  
pp. 79-88 ◽  
Author(s):  
Omar Zaidan
Keyword(s):  
Open Source ◽  
Machine Translation ◽  
Error Rate ◽  
Software Tool ◽  
Minimum Error ◽  
Open Source Tool ◽  
Series Of Experiments ◽  
Evaluation Metric ◽  
Minimum Error Rate Training ◽  
Translation Systems

Z-MERT: A Fully Configurable Open Source Tool for Minimum Error Rate Training of Machine Translation Systems We introduce Z-MERT, a software tool for minimum error rate training of machine translation systems (Och, 2003). In addition to being an open source tool that is extremely easy to compile and run, Z-MERT is also agnostic regarding the evaluation metric, fully configurable, and requires no modification to work with any decoder. We describe Z-MERT and review its features, and report the results of a series of experiments that examine the tool's runtime. We establish that Z-MERT is extremely efficient, making it well-suited for time-sensitive pipelines. The experiments also provide an insight into the tool's runtime in terms of several variables (size of the development set, size of produced N-best lists, etc).

Machine Translation Evaluation: Unveiling the Role of Dense Sentence Vector Embedding for Morphologically Rich Language

2019 ◽  
Vol 34 (01) ◽  
pp. 2059001
Author(s):  
Samiksha Tripathi ◽  
Vineet Kansal
Keyword(s):  
Machine Translation ◽  
Optimal Solution ◽  
Poor Performance ◽  
Target Language ◽  
Linguistic Knowledge ◽  
Machine Translation Evaluation ◽  
Mt Evaluation ◽  
Morphologically Rich Languages ◽  
Evaluation Metric

Machine Translation (MT) evaluation metrics like BiLingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit Ordering (METEOR) are known to have poor performance for word-order and morphologically rich languages. Application of linguistic knowledge to evaluate MTs for morphologically rich language like Hindi as a target language, is shown to be more effective and accurate [S. Tripathi and V. Kansal, Using linguistic knowledge for machine translation evaluation with Hindi as a target language, Comput. Sist.21(4) (2017) 717–724]. Leveraging the recent progress made in the domain of word vector and sentence vector embedding [T. Mikolov and J. Dean, Distributed representations of words and phrases and their compositionality, Adv. Neural Inf. Process. Syst. 2 (2013) 3111–3119], authors have trained a large corpus of pre-processed Hindi text ([Formula: see text] million tokens) for obtaining the word vectors and sentence vector embedding for Hindi. The training has been performed on high end system configuration utilizing Google Cloud platform resources. This sentence vector embedding is further used to corroborate the findings through linguistic knowledge in evaluation metric. For morphologically rich language as target, evaluation metric of MT systems is considered as an optimal solution. In this paper, authors have demonstrated that MT evaluation using sentence embedding-based approach closely mirrors linguistic evaluation technique. The relevant codes used to generate the vector embedding for Hindi have been uploaded on code sharing platform Github. a

scholarly journals Designing a Frame-Semantic Machine Translation Evaluation Metric

2019 ◽  
Author(s):  
Oliver Czulo ◽  
◽  
Tiago Timponi Torrent ◽  
Ely Edison da Silva Matos ◽  
Alexandre Diniz da Costa ◽  
...  
Keyword(s):  
Machine Translation ◽  
Machine Translation Evaluation ◽  
Evaluation Metric

scholarly journals An Open-Source Web-Based Tool for Resource-Agnostic Interactive Translation Prediction

2014 ◽  
Vol 102 (1) ◽  
pp. 69-80 ◽  
Author(s):  
Torregrosa Daniel ◽  
Forcada Mikel L. ◽  
Pérez-Ortiz Juan Antonio
Keyword(s):  
Open Source ◽  
Machine Translation ◽  
Web Application ◽  
Statistical Machine Translation ◽  
Black Box ◽  
Translation System ◽  
Web Tool ◽  
Web Based ◽  
Strongly Coupled ◽  
Machine Translation System

Abstract We present a web-based open-source tool for interactive translation prediction (ITP) and describe its underlying architecture. ITP systems assist human translators by making context-based computer-generated suggestions as they type. Most of the ITP systems in literature are strongly coupled with a statistical machine translation system that is conveniently adapted to provide the suggestions. Our system, however, follows a resource-agnostic approach and suggestions are obtained from any unmodified black-box bilingual resource. This paper reviews our ITP method and describes the architecture of Forecat, a web tool, partly based on the recent technology of web components, that eases the use of our ITP approach in any web application requiring this kind of translation assistance. We also evaluate the performance of our method when using an unmodified Moses-based statistical machine translation system as the bilingual resource.

Sign in / Sign up

Export Citation Format

Share Document