Beyond MT metrics in specialised translation: Automated and manual evaluation of machine translation output for freelance translators and small LSPs in the context of EU documents

This paper discusses simplified methods of translation evaluation in two seemingly disparate areas: machine translation (MT) technology and translation for EU institutions. It provides a brief overview of methods for evaluating MT output and proposes simplified solutions for small LSPs and freelancers dealing with specialised translation of this kind. After discussing the context of the study and the process of machine translation, an analysis of fragments of the selected specialist text (an EU regulation) is carried out. The official English and Polish versions of this document provide the basis for a comparative evaluation of raw machine translation output obtained with selected commercially available (paid) neural machine translation engines (NMT). Quantitative analysis, including the Damerau-Levenshstein edit distance parameters and the number of erroneous segments in the text, combined with a manual qualitative analysis of errors and terminology can be a serviceable method for small LSPs and freelance translators to evaluate the usefulness of neural machine translation engines.