Extraction of multi-word expressions from small parallel corpora

2012 ◽  
Vol 18 (4) ◽  
pp. 549-573 ◽  
Author(s):  
YULIA TSVETKOV ◽  
SHULY WINTNER
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Simple Algorithm ◽  
Search Space ◽  
Translation System ◽  
Parallel Corpora ◽  
Source Language ◽  
Extraction Algorithm ◽  
Machine Translation System

AbstractWe present a general, novel methodology for extracting multi-word expressions (MWEs) of various types, along with their translations, from small, word-aligned parallel corpora. Unlike existing approaches, we focus on misalignments; these typically indicate expressions in the source language that are translated to the target in a non-compositional way. We introduce a simple algorithm that proposes MWE candidates based on such misalignments, relying on 1:1 alignments as anchors that delimit the search space. We use a large monolingual corpus to rank and filter these candidates. Evaluation of the quality of the extraction algorithm reveals significant improvements over naïve alignment-based methods. The extracted MWEs, with their translations, are used in the training of a statistical machine translation system, showing a small but significant improvement in its performance.

scholarly journals Developing Statistical Machine Translation System for English and Nigerian Languages

2018 ◽  
pp. 1-8
Author(s):  
Ignatius Ikechukwu Ayogu ◽  
Adebayo Olusola Adetunmbi ◽  
Bolanle Adefowoke Ojokoh
Keyword(s):  
Error Analysis ◽  
Machine Translation ◽  
Statistical Machine Translation ◽  
Translation System ◽  
Parallel Corpora ◽  
Translation Tools ◽  
Global Demand ◽  
Machine Translation System ◽  
Translation Systems

The global demand for translation and translation tools currently surpasses the capacity of available solutions. Besides, there is no one-solution-fits-all, off-the-shelf solution for all languages. Thus, the need and urgency to increase the scale of research for the development of translation tools and devices continue to grow, especially for languages suffering under the pressure of globalisation. This paper discusses our experiments on translation systems between English and two Nigerian languages: Igbo and Yorùbá. The study is setup to build parallel corpora, train and experiment English-to-Igbo, (), English-to-Yorùbá, () and Igbo-to-Yorùbá, () phrase-based statistical machine translation systems. The systems were trained on parallel corpora that were created for each language pair using text from the religious domain in the course of this research. A BLEU score of 30.04, 29.01 and 18.72 respectively was recorded for the English-to-Igbo, English-to-Yorùbá and Igbo-to-Yorùbá MT systems. An error analysis of the systems’ outputs was conducted using a linguistically motivated MT error analysis approach and it showed that errors occurred mostly at the lexical, grammatical and semantic levels. While the study reveals the potentials of our corpora, it also shows that the size of the corpora is yet an issue that requires further attention. Thus an important target in the immediate future is to increase the quantity and quality of the data.  

scholarly journals Improving Machine Translation Performance by Exploiting Non-Parallel Corpora

2005 ◽  
Vol 31 (4) ◽  
pp. 477-504 ◽  
Author(s):  
Dragos Stefan Munteanu ◽  
Daniel Marcu
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Translation System ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Scarce Resources ◽  
Parallel Data ◽  
Machine Translation System ◽  
Novel Method ◽  
Arabic And English

We present a novel method for discovering parallel sentences in comparable, non-parallel corpora. We train a maximum entropy classifier that, given a pair of sentences, can reliably determine whether or not they are translations of each other. Using this approach, we extract parallel data from large Chinese, Arabic, and English non-parallel newspaper corpora. We evaluate the quality of the extracted data by showing that it improves the performance of a state-of-the-art statistical machine translation system. We also show that a good-quality MT system can be built from scratch by starting with a very small parallel corpus (100,000 words) and exploiting a large non-parallel corpus. Thus, our method can be applied with great benefit to language pairs for which only scarce resources are available.

scholarly journals Predicting and Using a Pragmatic Component of Lexical Aspect

2016 ◽  
Vol 13 ◽  
Author(s):  
Sharid Loáiciga ◽  
Cristina Grisot
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Target Language ◽  
Translation System ◽  
Automatic Annotation ◽  
Parallel Corpora ◽  
Linguistic Data ◽  
Lexical Aspect ◽  
Machine Translation System ◽  
Automatic Systems

This paper proposes a method for improving the results of a statistical Machine Translation system using boundedness, a pragmatic component of the verbal phrase’s lexical aspect. First, the paper presents manual and automatic annotation experiments for lexical aspect in English-French parallel corpora. It will be shown that this aspectual property is identified and classified with ease both by humans and by automatic systems. Second, Statistical Machine Translation experiments using the boundedness annotations are presented. These experiments show that the information regarding lexical aspect is useful to improve the output of a Machine Translation system in terms of better choices of verbal tenses in the target language, as well as better lexical choices. Ultimately, this work aims at providing a method for the automatic annotation of data with boundedness information and at contributing to Machine Translation by taking into account linguistic data.

scholarly journals Exploration of Corpus Augmentation Approach for English-Hindi Bidirectional Statistical Machine Translation System

2016 ◽  
Vol 6 (3) ◽  
pp. 1059 ◽  
Author(s):  
K. Jaya ◽  
Deepa Gupta
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Research Work ◽  
Translation System ◽  
Indian Languages ◽  
Translation Quality ◽  
Parallel Data ◽  
Machine Translation System ◽  
Best Parameter

Even though lot of Statistical Machine Translation(SMT) research work is happening for English-Hindi language pair, there is no effort done to standardize the dataset. Each of the research work uses different dataset, different parameters and different number of sentences during various phases of translation resulting in varied translation output. So comparing  these models, understand the result of these models, to get insight into corpus behavior for these models, regenerating the result of these research work  becomes tedious. This necessitates the need for standardization of dataset and to identify the common parameter for the development of model.  The main contribution of this paper is to discuss an approach to standardize the dataset and to identify the best parameter which in combination gives best performance. It also investigates a novel corpus augmentation approach to improve the translation quality of English-Hindi bidirectional statistical machine translation system. This model works well for the scarce resource without incorporating the external parallel data corpus of the underlying language.  This experiment is carried out using Open Source phrase-based toolkit Moses. Indian Languages Corpora Initiative (ILCI) Hindi-English tourism corpus is used.  With limited dataset, considerable improvement is achieved using the corpus augmentation approach for the English-Hindi bidirectional SMT system.

Exploration of Corpus Augmentation Approach for English-Hindi Bidirectional Statistical Machine Translation System

2016 ◽  
Vol 6 (3) ◽  
pp. 1059 ◽  
Author(s):  
K. Jaya ◽  
Deepa Gupta
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Research Work ◽  
Translation System ◽  
Indian Languages ◽  
Translation Quality ◽  
Parallel Data ◽  
Machine Translation System ◽  
Best Parameter

Even though lot of Statistical Machine Translation(SMT) research work is happening for English-Hindi language pair, there is no effort done to standardize the dataset. Each of the research work uses different dataset, different parameters and different number of sentences during various phases of translation resulting in varied translation output. So comparing  these models, understand the result of these models, to get insight into corpus behavior for these models, regenerating the result of these research work  becomes tedious. This necessitates the need for standardization of dataset and to identify the common parameter for the development of model.  The main contribution of this paper is to discuss an approach to standardize the dataset and to identify the best parameter which in combination gives best performance. It also investigates a novel corpus augmentation approach to improve the translation quality of English-Hindi bidirectional statistical machine translation system. This model works well for the scarce resource without incorporating the external parallel data corpus of the underlying language.  This experiment is carried out using Open Source phrase-based toolkit Moses. Indian Languages Corpora Initiative (ILCI) Hindi-English tourism corpus is used.  With limited dataset, considerable improvement is achieved using the corpus augmentation approach for the English-Hindi bidirectional SMT system.

English-Dogri Translation System using MOSES

2016 ◽  
Vol 1 (1) ◽  
pp. 45-49
Author(s):  
Avinash Singh ◽  
Asmeet Kour ◽  
Shubhnandan S. Jamwal
Keyword(s):  
Natural Language Processing ◽  
Machine Translation ◽  
Language Processing ◽  
Statistical Machine Translation ◽  
Translation System ◽  
Parallel Corpus ◽  
English System ◽  
Machine Translation System ◽  
Translation Machine ◽  
Language Pair

The objective behind this paper is to analyze the English-Dogri parallel corpus translation. Machine translation is the translation from one language into another language. Machine translation is the biggest application of the Natural Language Processing (NLP). Moses is statistical machine translation system allow to train translation models for any language pair. We have developed translation system using Statistical based approach which helps in translating English to Dogri and vice versa. The parallel corpus consists of 98,973 sentences. The system gives accuracy of 80% in translating English to Dogri and the system gives accuracy of 87% in translating Dogri to English system.

scholarly journals An Open-Source Web-Based Tool for Resource-Agnostic Interactive Translation Prediction

2014 ◽  
Vol 102 (1) ◽  
pp. 69-80 ◽  
Author(s):  
Torregrosa Daniel ◽  
Forcada Mikel L. ◽  
Pérez-Ortiz Juan Antonio
Keyword(s):  
Open Source ◽  
Machine Translation ◽  
Web Application ◽  
Statistical Machine Translation ◽  
Black Box ◽  
Translation System ◽  
Web Tool ◽  
Web Based ◽  
Strongly Coupled ◽  
Machine Translation System

Abstract We present a web-based open-source tool for interactive translation prediction (ITP) and describe its underlying architecture. ITP systems assist human translators by making context-based computer-generated suggestions as they type. Most of the ITP systems in literature are strongly coupled with a statistical machine translation system that is conveniently adapted to provide the suggestions. Our system, however, follows a resource-agnostic approach and suggestions are obtained from any unmodified black-box bilingual resource. This paper reviews our ITP method and describes the architecture of Forecat, a web tool, partly based on the recent technology of web components, that eases the use of our ITP approach in any web application requiring this kind of translation assistance. We also evaluate the performance of our method when using an unmodified Moses-based statistical machine translation system as the bilingual resource.

Development of a method for assessing the quality of machine translation based on ensemble methods in machine learning

2021 ◽  
Author(s):  
A.V. Kozina ◽  
Yu.S. Belov
Keyword(s):  
Machine Learning ◽  
Random Forest ◽  
Quality Assessment ◽  
Machine Translation ◽  
Translation System ◽  
Human Judgment ◽  
Translation Quality ◽  
Machine Translation System ◽  
Translation Systems

Automatically assessing the quality of machine translation is an important yet challenging task for machine translation research. Translation quality assessment is understood as predicting translation quality without reference to the source text. Translation quality depends on the specific machine translation system and often requires post-editing. Manual editing is a long and expensive process. Since the need to quickly determine the quality of translation increases, its automation is required. In this paper, we propose a quality assessment method based on ensemble supervised machine learning methods. The bilingual corpus WMT 2019 for the EnglishRussian language pair was used as data. The text data volume is 17089 sentences, 85% of the data was used for training, and 15% for testing the model. Linguistic functions extracted from the text in the source and target languages were used as features for training the system, since it is these characteristics that can most accurately characterize the translation in terms of quality. The following tools were used for feature extraction: a free language modeling tool based on SRILM and a Stanford POS Tagger parts of speech tagger. Before training the system, the text was preprocessed. The model was trained using three regression methods: Bagging, Extra Tree, and Random Forest. The algorithms were implemented in the Python programming language using the Scikit learn library. The parameters of the random forest method have been optimized using a grid search. The performance of the model was assessed by the mean absolute error MAE and the root mean square error RMSE, as well as by the Pearsоn coefficient, which determines the correlation with human judgment. Testing was carried out using three machine translation systems: Google and Bing neural systems, Mouses statistical machine translation systems based on phrases and based on syntax. Based on the results of the work, the method of additional trees showed itself best. In addition, for all categories of indicators under consideration, the best results are achieved using the Google machine translation system. The developed method showed good results close to human judgment. The system can be used for further research in the task of assessing the quality of translation.

Sign in / Sign up

Export Citation Format

Share Document