Unsupervised Machine Translation Quality Estimation in Black-Box Setting

Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation, and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By utilizing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivaling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.

Download Full-text

Online Multitask Learning for Machine Translation Quality Estimation

10.3115/v1/p15-1022 ◽

2015 ◽

Author(s):

José G. C. de Souza ◽

Matteo Negri ◽

Elisa Ricci ◽

Marco Turchi

Keyword(s):

Machine Translation ◽

Multitask Learning ◽

Quality Estimation ◽

Translation Quality

Download Full-text

Machine Translation Quality Estimation: Applications and Future Perspectives

Machine Translation: Technologies and Applications - Translation Quality Assessment ◽

10.1007/978-3-319-91241-7_10 ◽

2018 ◽

pp. 201-235 ◽

Cited By ~ 1

Author(s):

Lucia Specia ◽

Kashif Shah

Keyword(s):

Machine Translation ◽

Future Perspectives ◽

Quality Estimation ◽

Translation Quality

Download Full-text

Unsupervised Translation Quality Estimation for Digital Entertainment Content Subtitles

International Journal of Semantic Computing ◽

10.1142/s1793351x20500026 ◽

2020 ◽

Vol 14 (01) ◽

pp. 137-151

Author(s):

Prabhakar Gupta ◽

Mayank Sharma

Keyword(s):

Machine Translation ◽

Experimental Results ◽

Parallel Translation ◽

Word Embeddings ◽

Quality Estimation ◽

Neural Machine Translation ◽

Intensity Index ◽

Translation Quality ◽

Digital Entertainment ◽

Reference Corpus

We demonstrate the potential for using aligned bilingual word embeddings in developing an unsupervised method to evaluate machine translations without a need for parallel translation corpus or reference corpus. We explain different aspects of digital entertainment content subtitles. We share our experimental results for four languages pairs English to French, German, Portuguese, Spanish, and present findings on the shortcomings of Neural Machine Translation for subtitles. We propose several improvements over the system designed by Gupta et al. [P. Gupta, S. Shekhawat and K. Kumar, Unsupervised quality estimation without reference corpus for subtitle machine translation using word embeddings, IEEE 13th Int. Conf. Semantic Computing, 2019, pp. 32–38.] by incorporating custom embedding model curated to subtitles, compound word splits and punctuation inclusion. We show a massive run time improvement of the order of [Formula: see text] by considering three types of edits, removing Proximity Intensity Index (PII) and changing post-edit score calculation from their system.

Download Full-text

QuEst – Design, Implementation and Extensions of a Framework for Machine Translation Quality Estimation

Prague Bulletin of Mathematical Linguistics ◽

10.2478/pralin-2013-0008 ◽

2013 ◽

Vol 100 (1) ◽

Cited By ~ 5

Author(s):

Kashif Shah ◽

Eleftherios Avramidis ◽

Ergun Biçici ◽

Lucia Specia

Keyword(s):

Machine Translation ◽

Quality Estimation ◽

Translation Quality

Download Full-text

The Role of Machine Translation Quality Estimation in the Post-Editing Workflow

Informatics ◽

10.3390/informatics8030061 ◽

2021 ◽

Vol 8 (3) ◽

pp. 61

Author(s):

Hannah Béchara ◽

Constantin Orăsan ◽

Carla Parra Escartín ◽

Marcos Zampieri ◽

William Lowe

Keyword(s):

Cognitive Load ◽

Machine Translation ◽

Impact Study ◽

Quality Estimation ◽

Editing Efficiency ◽

Translation Quality ◽

Automatic Translation ◽

New Challenges

As Machine Translation (MT) becomes increasingly ubiquitous, so does its use in professional translation workflows. However, its proliferation in the translation industry has brought about new challenges in the field of Post-Editing (PE). We are now faced with a need to find effective tools to assess the quality of MT systems to avoid underpayments and mistrust by professional translators. In this scenario, one promising field of study is MT Quality Estimation (MTQE), as this aims to determine the quality of an automatic translation and, indirectly, its degree of post-editing difficulty. However, its impact on the translation workflows and the translators’ cognitive load is still to be fully explored. We report on the results of an impact study engaging professional translators in PE tasks using MTQE. To assess the translators’ cognitive load we measure their productivity both in terms of time and effort (keystrokes) in three different scenarios: translating from scratch, post-editing without using MTQE, and post-editing using MTQE. Our results show that good MTQE information can improve post-editing efficiency and decrease the cognitive load on translators. This is especially true for cases with low MT quality.

Download Full-text