Towards Automatic Error Analysis of Machine Translation Output

Evaluation and error analysis of machine translation output are important but difficult tasks. In this article, we propose a framework for automatic error analysis and classification based on the identification of actual erroneous words using the algorithms for computation of Word Error Rate (WER) and Position-independent word Error Rate (PER), which is just a very first step towards development of automatic evaluation measures that provide more specific information of certain translation problems. The proposed approach enables the use of various types of linguistic knowledge in order to classify translation errors in many different ways. This work focuses on one possible set-up, namely, on five error categories: inflectional errors, errors due to wrong word order, missing words, extra words, and incorrect lexical choices. For each of the categories, we analyze the contribution of various POS classes. We compared the results of automatic error analysis with the results of human error analysis in order to investigate two possible applications: estimating the contribution of each error type in a given translation output in order to identify the main sources of errors for a given translation system, and comparing different translation outputs using the introduced error categories in order to obtain more information about advantages and disadvantages of different systems and possibilites for improvements, as well as about advantages and disadvantages of applied methods for improvements. We used Arabic–English Newswire and Broadcast News and Chinese–English Newswire outputs created in the framework of the GALE project, several Spanish and English European Parliament outputs generated during the TC-Star project, and three German–English outputs generated in the framework of the fourth Machine Translation Workshop. We show that our results correlate very well with the results of a human error analysis, and that all our metrics except the extra words reflect well the differences between different versions of the same translation system as well as the differences between different translation systems.

Download Full-text

Research on Machine Translation System of Semantic Network Partition

Advanced Materials Research ◽

10.4028/www.scientific.net/amr.546-547.1340 ◽

2012 ◽

Vol 546-547 ◽

pp. 1340-1344

Author(s):

Hui Ming Su

Keyword(s):

Machine Translation ◽

Question Answering ◽

Semantic Network ◽

Translation System ◽

Library Screening ◽

Network Partition ◽

Syntax Analysis ◽

Question Answering System ◽

Advantages And Disadvantages ◽

Machine Translation System

This paper reports a new architecture of intelligent question answering system based on the analysis of answering systems’ advantages and disadvantages. According to the design of ARIANE system, it develops the machine translation system for problem extraction. In the machine translation system design, analysis, and intelligent syntax has been difficult to solve. For the syntax analysis and word construction problems in intelligent segmentation, this paper adopts semantic network partition to solve it, It is partition in some level of association, can batter handle the issue to focus in semantic network partition. In this way, the user issues after word from the words library screening, a list of a series of related issues in descending order by degree of difficulty and returned to the user. Experimental analysis proves the usability of the improved algorithm.

Download Full-text

Development of English-Hindi Interactive Machine Translation

International Journal of Engineering and Advanced Technology - Regular Issue ◽

10.35940/ijeat.a1532.109119 ◽

2019 ◽

Vol 9 (1) ◽

pp. 4185-4189

Keyword(s):

Machine Translation ◽

Error Rate ◽

Statistical Machine Translation ◽

Indian Languages ◽

Word Error Rate ◽

Translation Error ◽

Translation Systems

Machine Translation systems are still far from being perfect and to improve their performance the concept of Interactive Machine Translation (IMT) was introduced. This paper proposes an IMT system, which uses Statistical Machine Translation and a bilingual corpus on which several algorithms (Word error rate, Position Independent Error Rate, Translation Error Rate, n-grams) are implemented to translate text from English to Indian languages. The proposed system improves both the speed and productivity of the human translators as found through experiments.

Download Full-text

Developing Statistical Machine Translation System for English and Nigerian Languages

Asian Journal of Research in Computer Science ◽

10.9734/ajrcos/2018/v1i424761 ◽

2018 ◽

pp. 1-8

Author(s):

Ignatius Ikechukwu Ayogu ◽

Adebayo Olusola Adetunmbi ◽

Bolanle Adefowoke Ojokoh

Keyword(s):

Error Analysis ◽

Machine Translation ◽

Statistical Machine Translation ◽

Translation System ◽

Parallel Corpora ◽

Translation Tools ◽

Global Demand ◽

Machine Translation System ◽

Translation Systems

The global demand for translation and translation tools currently surpasses the capacity of available solutions. Besides, there is no one-solution-fits-all, off-the-shelf solution for all languages. Thus, the need and urgency to increase the scale of research for the development of translation tools and devices continue to grow, especially for languages suffering under the pressure of globalisation. This paper discusses our experiments on translation systems between English and two Nigerian languages: Igbo and Yorùbá. The study is setup to build parallel corpora, train and experiment English-to-Igbo, (), English-to-Yorùbá, () and Igbo-to-Yorùbá, () phrase-based statistical machine translation systems. The systems were trained on parallel corpora that were created for each language pair using text from the religious domain in the course of this research. A BLEU score of 30.04, 29.01 and 18.72 respectively was recorded for the English-to-Igbo, English-to-Yorùbá and Igbo-to-Yorùbá MT systems. An error analysis of the systems’ outputs was conducted using a linguistically motivated MT error analysis approach and it showed that errors occurred mostly at the lexical, grammatical and semantic levels. While the study reveals the potentials of our corpora, it also shows that the size of the corpora is yet an issue that requires further attention. Thus an important target in the immediate future is to increase the quantity and quality of the data.

Download Full-text

OCR Challenges for a Latvian Pronunciation Dictionary

Frontiers in Artificial Intelligence and Applications - Human Language Technologies – The Baltic Perspective ◽

10.3233/faia200623 ◽

2020 ◽

Author(s):

Laine Strankale ◽

Pēteris Paikens

Keyword(s):

Error Analysis ◽

Open Source ◽

Error Rate ◽

Post Processing ◽

Speech Technology ◽

Word Error Rate ◽

Additional Support ◽

Further Development ◽

Pronunciation Dictionary

This paper covers the devlopment of a custom OCR solution based on the Tesseract open source engine developed for digitization of a Latvian pronunciation dictionary where the pronunciation data is described using a large variety of diacritic markings not supported by standard OCR solutions. We describe our efforts in training a model for these symbols without the additional support of preexisting dictionaries and illustrate how word error rate (WER) and character error rate (CER) are affected by changes in the dataset content and size. We also provide an error analysis and postulate possible causes for common pitfalls. The resulting model achieved a CER of 2.07%, making it suitable for digitization of the whole dictionary in combination with heuristic post-processing and proofreading, resulting in a useful resource for further development of speech technology for Latvian.

Download Full-text

Emerging trends: APIs for speech and machine translation and more

Natural Language Engineering ◽

10.1017/s1351324918000335 ◽

2018 ◽

Vol 24 (6) ◽

pp. 951-960 ◽

Cited By ~ 1

Author(s):

KENNETH WARD CHURCH

Keyword(s):

Machine Translation ◽

Error Rate ◽

Word Error Rate ◽

Wrong Thing ◽

Emerging Trends ◽

What Matters

AbstractLots of companies are offering lots of APIs. Reviews are not always as constructive as they could be. Some reviews encourage unproductive work on checkbox features that no one wants. It makes no sense to do the wrong thing badly. Constructive reviews should help focus priorities on what matters. Users care more about a great box opening experience than small improvements in word error rate and BLEU, popular metrics for speech and translation. In 15 minutes or less, can we teach potential users something new (and fun) that does something useful, such as how to translate PowerPoint between English and Chinese, preserving many of the features that are important to PowerPoint such as graphics and animations? See Appendix for the solution.

Download Full-text

Morpho-syntactic information for automatic error analysis of statistical machine translation output

10.3115/1654650.1654652 ◽

2006 ◽

Cited By ~ 6

Author(s):

Maja Popović ◽

Hermann Ney ◽

Adrià de Gispert ◽

José B. Mariño ◽

Deepa Gupta ◽

...

Keyword(s):

Error Analysis ◽

Machine Translation ◽

Statistical Machine Translation ◽

Syntactic Information ◽

Automatic Error

Download Full-text

Machine translation of English speech: Comparison of multiple algorithms

Journal of Intelligent Systems ◽

10.1515/jisys-2022-0005 ◽

2022 ◽

Vol 31 (1) ◽

pp. 159-167

Author(s):

Yijun Wu ◽

Yonghong Qin

Keyword(s):

Neural Network ◽

Machine Translation ◽

Error Rate ◽

Short Term Memory ◽

Back Propagation ◽

Word Error Rate ◽

The Neural Network ◽

Actual Speech ◽

Neural Network Algorithm ◽

Translation Machine

Abstract In order to improve the efficiency of the English translation, machine translation is gradually and widely used. This study briefly introduces the neural network algorithm for speech recognition. Long short-term memory (LSTM), instead of traditional recurrent neural network (RNN), was used as the encoding algorithm for the encoder, and RNN as the decoding algorithm for the decoder. Then, simulation experiments were carried out on the machine translation algorithm, and it was compared with two other machine translation algorithms. The results showed that the back-propagation (BP) neural network had a lower word error rate and spent less recognition time than artificial recognition in recognizing the speech; the LSTM–RNN algorithm had a lower word error rate than BP–RNN and RNN–RNN algorithms in recognizing the test samples. In the actual speech translation test, as the length of speech increased, the LSTM–RNN algorithm had the least changes in the translation score and word error rate, and it had the highest translation score and the lowest word error rate under the same speech length.

Download Full-text

Perceptual Characteristics of Consonant Production in Apraxia of Speech and Aphasia

American Journal of Speech-Language Pathology ◽

10.1044/2019_ajslp-18-0169 ◽

2019 ◽

Vol 28 (4) ◽

pp. 1411-1431 ◽

Cited By ~ 3

Author(s):

Lauren Bislick ◽

William D. Hula

Keyword(s):

Error Rate ◽

Error Rates ◽

Group Differences ◽

Diagnostic Process ◽

Apraxia Of Speech ◽

Contextual Variables ◽

Error Type ◽

Type Distribution ◽

Phonetic Features ◽

Syllable Position

Purpose This retrospective analysis examined group differences in error rate across 4 contextual variables (clusters vs. singletons, syllable position, number of syllables, and articulatory phonetic features) in adults with apraxia of speech (AOS) and adults with aphasia only. Group differences in the distribution of error type across contextual variables were also examined. Method Ten individuals with acquired AOS and aphasia and 11 individuals with aphasia participated in this study. In the context of a 2-group experimental design, the influence of 4 contextual variables on error rate and error type distribution was examined via repetition of 29 multisyllabic words. Error rates were analyzed using Bayesian methods, whereas distribution of error type was examined via descriptive statistics. Results There were 4 findings of robust differences between the 2 groups. These differences were found for syllable position, number of syllables, manner of articulation, and voicing. Group differences were less robust for clusters versus singletons and place of articulation. Results of error type distribution show a high proportion of distortion and substitution errors in speakers with AOS and a high proportion of substitution and omission errors in speakers with aphasia. Conclusion Findings add to the continued effort to improve the understanding and assessment of AOS and aphasia. Several contextual variables more consistently influenced breakdown in participants with AOS compared to participants with aphasia and should be considered during the diagnostic process. Supplemental Material https://doi.org/10.23641/asha.9701690

Download Full-text