BiMEANT: Integrating Cross-Lingual and Monolingual Semantic Frame Similarities in the MEANT Semantic MT Evaluation Metric

2014 ◽  
pp. 82-93
Author(s):  
Chi-kiu Lo ◽  
Dekai Wu
Keyword(s):  
Mt Evaluation ◽  
Evaluation Metric ◽  
Cross Lingual

Machine Translation Evaluation: Unveiling the Role of Dense Sentence Vector Embedding for Morphologically Rich Language

2019 ◽  
Vol 34 (01) ◽  
pp. 2059001
Author(s):  
Samiksha Tripathi ◽  
Vineet Kansal
Keyword(s):  
Machine Translation ◽  
Optimal Solution ◽  
Poor Performance ◽  
Target Language ◽  
Linguistic Knowledge ◽  
Machine Translation Evaluation ◽  
Mt Evaluation ◽  
Morphologically Rich Languages ◽  
Evaluation Metric

Machine Translation (MT) evaluation metrics like BiLingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit Ordering (METEOR) are known to have poor performance for word-order and morphologically rich languages. Application of linguistic knowledge to evaluate MTs for morphologically rich language like Hindi as a target language, is shown to be more effective and accurate [S. Tripathi and V. Kansal, Using linguistic knowledge for machine translation evaluation with Hindi as a target language, Comput. Sist.21(4) (2017) 717–724]. Leveraging the recent progress made in the domain of word vector and sentence vector embedding [T. Mikolov and J. Dean, Distributed representations of words and phrases and their compositionality, Adv. Neural Inf. Process. Syst. 2 (2013) 3111–3119], authors have trained a large corpus of pre-processed Hindi text ([Formula: see text] million tokens) for obtaining the word vectors and sentence vector embedding for Hindi. The training has been performed on high end system configuration utilizing Google Cloud platform resources. This sentence vector embedding is further used to corroborate the findings through linguistic knowledge in evaluation metric. For morphologically rich language as target, evaluation metric of MT systems is considered as an optimal solution. In this paper, authors have demonstrated that MT evaluation using sentence embedding-based approach closely mirrors linguistic evaluation technique. The relevant codes used to generate the vector embedding for Hindi have been uploaded on code sharing platform Github. a

scholarly journals A simple automatic MT evaluation metric

2009 ◽  
Author(s):  
Petr Homola ◽  
Vladislav Kuboň ◽  
Pavel Pecina
Keyword(s):  
Mt Evaluation ◽  
Evaluation Metric

scholarly journals Unsupervised Quality Estimation Model for English to German Translation and Its Application in Extensive Supervised Evaluation

2014 ◽  
Vol 2014 ◽  
pp. 1-12 ◽  
Author(s):  
Aaron L.-F. Han ◽  
Derek F. Wong ◽  
Lidia S. Chao ◽  
Liangye He ◽  
Yi Lu
Keyword(s):  
Rapid Development ◽  
Linguistic Features ◽  
Estimation Model ◽  
Linguistic Feature ◽  
Language Bias ◽  
Automatic Translation ◽  
Part Of Speech ◽  
Mt Evaluation ◽  
Evaluation Metric ◽  
Translation Systems

With the rapid development of machine translation (MT), the MT evaluation becomes very important to timely tell us whether the MT system makes any progress. The conventional MT evaluation methods tend to calculate the similarity between hypothesis translations offered by automatic translation systems and reference translations offered by professional translators. There are several weaknesses in existing evaluation metrics. Firstly, the designed incomprehensive factors result in language-bias problem, which means they perform well on some special language pairs but weak on other language pairs. Secondly, they tend to use no linguistic features or too many linguistic features, of which no usage of linguistic feature draws a lot of criticism from the linguists and too many linguistic features make the model weak in repeatability. Thirdly, the employed reference translations are very expensive and sometimes not available in the practice. In this paper, the authors propose an unsupervised MT evaluation metric using universal part-of-speech tagset without relying on reference translations. The authors also explore the performances of the designed metric on traditional supervised evaluation tasks. Both the supervised and unsupervised experiments show that the designed methods yield higher correlation scores with human judgments.

scholarly journals A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity

2019 ◽  
Author(s):  
Yoshinari Fujinuma ◽  
Jordan Boyd-Graber ◽  
Michael J. Paul
Keyword(s):  
Word Embeddings ◽  
Evaluation Metric ◽  
Cross Lingual

ENTF: An Entropy-Based MT Evaluation Metric

2017 ◽  
pp. 68-77
Author(s):  
Hui Yu ◽  
Weizhi Xu ◽  
Shouxun Lin ◽  
Qun Liu
Keyword(s):  
Mt Evaluation ◽  
Evaluation Metric

scholarly journals Guidance to Pre-tokeniztion for SacreBLEU: Meta-Evaluation in Korean

2022 ◽  
Author(s):  
Ahrii Kim ◽  
Jinhyun Kim
Keyword(s):  
Empirical Study ◽  
Automatic Evaluation ◽  
Human Judgment ◽  
Evaluation Data ◽  
Human Evaluation ◽  
Mt Evaluation ◽  
Evaluation Metric ◽  
Agglutinative Languages

SacreBLEU, by incorporating a text normalizing step in the pipeline, has been well-received as an automatic evaluation metric in recent years. With agglutinative languages such as Korean, however, the metric cannot provide a conceivable result without the help of customized pre-tokenization. In this regard, this paper endeavors to examine the influence of diversified pre-tokenization schemes –word, morpheme, character, and subword– on the aforementioned metric by performing a meta-evaluation with manually-constructed into-Korean human evaluation data. Our empirical study demonstrates that the correlation of SacreBLEU (to human judgment) fluctuates consistently by the token type. The reliability of the metric even deteriorates due to some tokenization, and MeCab is not an exception. Guiding through the proper usage of tokenizer for each metric, we stress the significance of a character level and the insignificance of a Jamo level in MT evaluation.

Sign in / Sign up

Export Citation Format

Share Document