Extraction of Bilingual Dictionary from Comparable Corpora for Resource Scarce Languages

2020 ◽  
Vol 17 (1) ◽  
pp. 54-60
Author(s):  
B. S. Sowmya Lakshmi ◽  
B. R. Shambhavi
Keyword(s):  
Indian Language ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Comparable Corpora ◽  
Bilingual Dictionary ◽  
Low Resource ◽  
Technological University ◽  
Language Corpus ◽  
Language Pair

Visvesvaraya Technological University, Belagavi, Karnataka, India One of the promising resources to extract dictionaries are said to be parallel corpora. Majority of the substantial works are based on parallel corpora, whereas for the resource scarce language pairs building a parallel corpus is a challenging task. To prevail over this issue, researchers found comparable corpora could be an alternative to extract dictionary. Proposed approach is to extract dictionary for a low resource language pair English and Kannada using comparable corpora obtained from Wikipedia dumps and corpus received from Indian Language Corpus Initiative (ILCI). Dictionary constructed comprises of both translation and transliteration entities with term level associations from English to Kannada. Resultant dictionary is of size 77545 tokens with precision score of 0.79. Proposed work is independent of language and could be expanded to other language pairs.

scholarly journals Pseudotext Injection and Advance Filtering of Low-Resource Corpus for Neural Machine Translation

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Michael Adjeisah ◽  
Guohua Liu ◽  
Douglas Omwenga Nyabuga ◽  
Richard Nuetey Nortey ◽  
Jinling Song
Keyword(s):  
Machine Translation ◽  
Language Processing ◽  
Training Data ◽  
Target Language ◽  
Similarity Metrics ◽  
Mahalanobis Distances ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Low Resource ◽  
Sentence Level

Scaling natural language processing (NLP) to low-resourced languages to improve machine translation (MT) performance remains enigmatic. This research contributes to the domain on a low-resource English-Twi translation based on filtered synthetic-parallel corpora. It is often perplexing to learn and understand what a good-quality corpus looks like in low-resource conditions, mainly where the target corpus is the only sample text of the parallel language. To improve the MT performance in such low-resource language pairs, we propose to expand the training data by injecting synthetic-parallel corpus obtained by translating a monolingual corpus from the target language based on bootstrapping with different parameter settings. Furthermore, we performed unsupervised measurements on each sentence pair engaging squared Mahalanobis distances, a filtering technique that predicts sentence parallelism. Additionally, we extensively use three different sentence-level similarity metrics after round-trip translation. Experimental results on a diverse amount of available parallel corpus demonstrate that injecting pseudoparallel corpus and extensive filtering with sentence-level similarity metrics significantly improves the original out-of-the-box MT systems for low-resource language pairs. Compared with existing improvements on the same original framework under the same structure, our approach exhibits tremendous developments in BLEU and TER scores.

scholarly journals Using ParaConc to extract bilingual terminology from parallel corpora: A case of English and Ndebele

Literator ◽  
2016 ◽  
Vol 37 (1) ◽  
Author(s):  
Ketiwe Ndhlovu
Keyword(s):  
Media Law ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
African Languages ◽  
Bilingual Dictionary ◽  
Bilingual Dictionaries ◽  
Key Word ◽  
Science And Education ◽  
Frequency Feature

The development of African languages into languages of science and technology is dependent on action being taken to promote the use of these languages in specialised fields such as technology, commerce, administration, media, law, science and education among others. One possible way of developing African languages is the compilation of specialised dictionaries (Chabata 2013). This article explores how parallel corpora can be interrogated using a bilingual concordancer (ParaConc) to extract bilingual terminology that can be used to create specialised bilingual dictionaries. An English–Ndebele Parallel Corpus was used as a resource and through ParaConc, an alphabetic list was compiled from which headwords and possible translations were sought. These translations provided possible terms for entry in a bilingual dictionary. The frequency feature and ‘hot words’ tool in ParaConc were used to determine the suitability of terms for inclusion in the dictionary and for identifying possible synonyms, respectively. Since parallel corpora are aligned and data are presented in context (Key Word in Context), it was possible to draw examples showing how headwords are used. Using this approach produced results quickly and accurately, whilst minimising the process of translating terms manually. It was noted that the quality of the dictionary is dependent on the quality of the corpus, hence the need for creating a representative and clean corpus needs to be emphasised. Although technology has multiple benefits in dictionary making, the research underscores the importance of collaboration between lexicographers, translators, subject experts and target communities so that representative dictionaries are created.

scholarly journals Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

2019 ◽  
Vol 7 ◽  
pp. 597-610 ◽  
Author(s):  
Mikel Artetxe ◽  
Holger Schwenk
Keyword(s):  
Natural Language ◽  
Similarity Search ◽  
Document Classification ◽  
Data Set ◽  
Test Set ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Low Resource ◽  
Cross Lingual

We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared byte-pair encoding vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI data set), cross-lingual document classification (MLDoc data set), and parallel corpus mining (BUCC data set) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low- resource languages. Our implementation, the pre-trained encoder, and the multilingual test set are available at https://github.com/facebookresearch/LASER .

scholarly journals Creating a bilingual dictionary of collocations: A learner-oriented approach

2021 ◽  
Vol 10 (3) ◽  
Author(s):  
Sonia Vaupot
Keyword(s):  
Translation Studies ◽  
First Year ◽  
Teaching Methodology ◽  
Master's Students ◽  
Comparable Corpora ◽  
Bilingual Dictionary ◽  
Active Approach ◽  
Learning Path ◽  
Target Languages ◽  
Language Pair

Considering the lack of specialised dictionaries in certain fields, a creative way of teaching through corpora-based work was proposed in a seminar for master’s students of translation studies (University of Ljubljana, Slovenia). Since phraseology and terminology play an important role both in specialised translation and in the learning path of students of translation studies, this article presents an active approach aimed at creating an online lexicographic resource in languages for specific purposes by using the didactic tool and database ARTES (Aide à la Rédaction de TExtes Scientifiques/Dictionary-assisted writing tool for scientific communication) previously developed at the Université de Paris (France). About thirty Slovene students enrolled in the first year of master’s study have been participating in the bilateral project since 2018. The aims of such an activity are multiple: students learn in a practical way how to compile comparable corpora from the internet, using the online corpus software Sketch Engine, to find similar linguistic constructions in the source and target languages. They also learn to create an online bilingual phraseological and terminological dictionary to facilitate the translation of specialised texts. In this way, they acquire skills and develop some knowledge in translation, terminology, and discourse phraseology. The article first describes the ARTES online database. Then, we present the teaching methodology and the students’ work, which consists of compiling corpora, extracting and translating collocations for the language pair French-Slovene, and entering them in the ARTES database. Finally, we propose an analysis of the most frequent collocation structures in both languages. The language pair considered here is French and Slovene, but the methodology can be applied to any other language pair.

Parallel sentence extraction to improve cross-language information retrieval from Wikipedia

2021 ◽  
pp. 016555152199275
Author(s):  
Juryong Cheon ◽  
Youngjoong Ko
Keyword(s):  
Information Retrieval ◽  
Language Resources ◽  
Query Translation ◽  
Factors Affecting ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Bilingual Dictionary ◽  
Sentence Extraction ◽  
Cross Language Information Retrieval ◽  
Cross Language

Translation language resources, such as bilingual word lists and parallel corpora, are important factors affecting the effectiveness of cross-language information retrieval (CLIR) systems. In particular, when large domain-appropriate parallel corpora are not available, developing an effective CLIR system is particularly difficult. Furthermore, creating a large parallel corpus is costly and requires considerable effort. Therefore, we here demonstrate the construction of parallel corpora from Wikipedia as well as improved query translation, wherein the queries are used for a CLIR system. To do so, we first constructed a bilingual dictionary, termed WikiDic. Then, we evaluated individual language resources and combinations of them in terms of their ability to extract parallel sentences; the combinations of our proposed WikiDic with the translation probability from the Web’s bilingual example sentence pairs and WikiDic was found to be best suited to parallel sentence extraction. Finally, to evaluate the parallel corpus generated from this best combination of language resources, we compared its performance in query translation for CLIR to that of a manually created English–Korean parallel corpus. As a result, the corpus generated by our proposed method achieved a better performance than did the manually created corpus, thus demonstrating the effectiveness of the proposed method for automatic parallel corpus extraction. Not only can the method demonstrated herein be used to inform the construction of other parallel corpora from language resources that are readily available, but also, the parallel sentence extraction method will naturally improve as Wikipedia continues to be used and its content develops.

Semantic approach for building generated virtual-parallel corpora from monolingual texts

2019 ◽  
Vol 55 (2) ◽  
pp. 469-490
Author(s):  
Krzysztof Wołk ◽  
Agnieszka Wołk ◽  
Krzysztof Marasek
Keyword(s):  
Machine Translation ◽  
Language Processing ◽  
Research Process ◽  
Levenshtein Distance ◽  
Language Resources ◽  
Semantic Approach ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Comparable Corpora ◽  
Translation Systems

Abstract Several natural languages have undergone a great deal of processing, but the problem of limited textual linguistic resources remains. The manual creation of parallel corpora by humans is rather expensive and time consuming, while the language data required for statistical machine translation (SMT) do not exist in adequate quantities for their statistical information to be used to initiate the research process. On the other hand, applying known approaches to build parallel resources from multiple sources, such as comparable or quasi-comparable corpora, is very complicated and provides rather noisy output, which later needs to be further processed and requires in-domain adaptation. To optimize the performance of comparable corpora mining algorithms, it is essential to use a quality parallel corpus for training of a good data classifier. In this research, we have developed a methodology for generating an accurate parallel corpus (Czech-English, Polish-English) from monolingual resources by calculating the compatibility between the results of three machine translation systems. We have created translations of large, single-language resources by applying multiple translation systems and strictly measuring translation compatibility using rules based on the Levenshtein distance. The results produced by this approach were very favorable. The generated corpora successfully improved the quality of SMT systems and seem to be useful for many other natural language processing tasks.

Parallel Data Extraction using Word Embeddings

2020 ◽  
Author(s):  
Pintu Lohar ◽  
Andy Way
Keyword(s):  
Information Retrieval ◽  
Data Extraction ◽  
Training Data ◽  
User Generated Content ◽  
Data Sets ◽  
Word Embeddings ◽  
Parallel Corpus ◽  
Comparable Corpora ◽  
Bilingual Dictionary ◽  
Parallel Data

Building a robust MT system requires a sufficiently large parallel corpus to be available as training data. In this paper, we propose to automatically extract parallel sentences from comparable corpora without using any MT system or even any parallel corpus at all. Instead, we use crosslingual information retrieval (CLIR), average word embeddings, text similarity and a bilingual dictionary, thus saving a significant amount of time and effort as no MT system is involved in this process. We conduct experiments on two different kinds of data: (i) formal texts from news domain, and (ii) user-generated content (UGC) from hotel reviews. The automatically extracted sentence pairs are then added to the already available parallel training data and the extended translation models are built from the concatenated data sets. Finally, we compare the performance of our new extended models against the baseline models built from the available data. The experimental evaluation reveals that our proposed approach is capable of improving the translation outputs for both the formal texts and UGC.

scholarly journals Neural Machine Translation for Indian Languages

2019 ◽  
Vol 28 (3) ◽  
pp. 465-477 ◽  
Author(s):  
Amarnath Pathak ◽  
Partha Pakray
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Indian Languages ◽  
Indian Language ◽  
Neural Machine Translation ◽  
Parallel Corpora ◽  
Parallel Corpus ◽  
Linguistic Resources ◽  
Linguistic Backgrounds

Abstract Machine Translation bridges communication barriers and eases interaction among people having different linguistic backgrounds. Machine Translation mechanisms exploit a range of techniques and linguistic resources for translation prediction. Neural machine translation (NMT), in particular, seeks optimality in translation through training of neural network, using a parallel corpus having a considerable number of instances in the form of a parallel running source and target sentences. Easy availability of parallel corpora for major Indian language forms and the ability of NMT systems to better analyze context and produce fluent translation make NMT a prominent choice for the translation of Indian languages. We have trained, tested, and analyzed NMT systems for English to Tamil, English to Hindi, and English to Punjabi translations. Predicted translations have been evaluated using Bilingual Evaluation Understudy and by human evaluators to assess the quality of translation in terms of its adequacy, fluency, and correspondence with human-predicted translation.

English-Dogri Translation System using MOSES

2016 ◽  
Vol 1 (1) ◽  
pp. 45-49
Author(s):  
Avinash Singh ◽  
Asmeet Kour ◽  
Shubhnandan S. Jamwal
Keyword(s):  
Natural Language Processing ◽  
Machine Translation ◽  
Language Processing ◽  
Statistical Machine Translation ◽  
Translation System ◽  
Parallel Corpus ◽  
English System ◽  
Machine Translation System ◽  
Translation Machine ◽  
Language Pair

The objective behind this paper is to analyze the English-Dogri parallel corpus translation. Machine translation is the translation from one language into another language. Machine translation is the biggest application of the Natural Language Processing (NLP). Moses is statistical machine translation system allow to train translation models for any language pair. We have developed translation system using Statistical based approach which helps in translating English to Dogri and vice versa. The parallel corpus consists of 98,973 sentences. The system gives accuracy of 80% in translating English to Dogri and the system gives accuracy of 87% in translating Dogri to English system.

Sign in / Sign up

Export Citation Format

Share Document