Handling Unknown Words in Statistical Machine Translation from a New Perspective

2012 ◽  
pp. 176-187 ◽  
Author(s):  
Jiajun Zhang ◽  
Feifei Zhai ◽  
Chengqing Zong
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Unknown Words ◽  
New Perspective

scholarly journals Hindi Chhattisgarhi Machine Translation System Using Statistical Approach

Webology ◽  
2021 ◽  
Vol 18 (Special Issue 02) ◽  
pp. 208-222
Author(s):  
Vikas Pandey ◽  
Dr.M.V. Padmavati ◽  
Dr. Ramesh Kumar
Keyword(s):  
Machine Translation ◽  
Language Processing ◽  
Statistical Approach ◽  
Statistical Machine Translation ◽  
Target Language ◽  
Translation System ◽  
Parallel Corpus ◽  
Machine Translation System ◽  
Unknown Words ◽  
Language Pair

Machine Translation is a subfield of Natural language Processing (NLP) which uses to translate source language to target language. In this paper an attempt has been made to make a Hindi Chhattisgarhi machine translation system which is based on statistical approach. In the state of Chhattisgarh there is a long awaited need for Hindi to Chhattisgarhi machine translation system for converting Hindi into Chhattisgarhi especially for non Chhattisgarhi speaking people. In order to develop Hindi Chhattisgarhi statistical machine translation system an open source software called Moses is used. Moses is a statistical machine translation system and used to automatically train the translation model for Hindi Chhattisgarhi language pair called as parallel corpus. A collection of structured text to study linguistic properties is called corpus. This machine translation system works on parallel corpus of 40,000 Hindi-Chhattisgarhi bilingual sentences. In order to overcome translation problem related to proper noun and unknown words, a transliteration system is also embedded in it. These sentences are extracted from various domains like stories, novels, text books and news papers etc. This system is tested on 1000 sentences to check the grammatical correctness of sentences and it was found that an accuracy of 75% is achieved.

scholarly journals Improving Machine Translation through Linked Data

2017 ◽  
Vol 108 (1) ◽  
pp. 355-366 ◽  
Author(s):  
Ankit Srivastava ◽  
Georg Rehm ◽  
Felix Sasaki
Keyword(s):  
Machine Translation ◽  
Language Processing ◽  
Linked Data ◽  
Statistical Machine Translation ◽  
Open Data ◽  
Knowledge Bases ◽  
Data Sets ◽  
Lexical Resources ◽  
Unknown Words ◽  
Cross Lingual

Abstract With the ever increasing availability of linked multilingual lexical resources, there is a renewed interest in extending Natural Language Processing (NLP) applications so that they can make use of the vast set of lexical knowledge bases available in the Semantic Web. In the case of Machine Translation, MT systems can potentially benefit from such a resource. Unknown words and ambiguous translations are among the most common sources of error. In this paper, we attempt to minimise these types of errors by interfacing Statistical Machine Translation (SMT) models with Linked Open Data (LOD) resources such as DBpedia and BabelNet. We perform several experiments based on the SMT system Moses and evaluate multiple strategies for exploiting knowledge from multilingual linked data in automatically translating named entities. We conclude with an analysis of best practices for multilingual linked data sets in order to optimise their benefit to multilingual and cross-lingual applications.

scholarly journals Decoding Strategies for Improving Low-Resource Machine Translation

Electronics ◽  
2020 ◽  
Vol 9 (10) ◽  
pp. 1562
Author(s):  
Chanjun Park ◽  
Yeongwook Yang ◽  
Kinam Park ◽  
Heuiseok Lim
Keyword(s):  
Machine Translation ◽  
Language Processing ◽  
Data Augmentation ◽  
Model Performance ◽  
Post Processing ◽  
Low Resource ◽  
Processing Strategies ◽  
N Gram ◽  
Unknown Words ◽  
New Perspective

Pre-processing and post-processing are significant aspects of natural language processing (NLP) application software. Pre-processing in neural machine translation (NMT) includes subword tokenization to alleviate the problem of unknown words, parallel corpus filtering that only filters data suitable for training, and data augmentation to ensure that the corpus contains sufficient content. Post-processing includes automatic post editing and the application of various strategies during decoding in the translation process. Most recent NLP researches are based on the Pretrain-Finetuning Approach (PFA). However, when small and medium-sized organizations with insufficient hardware attempt to provide NLP services, throughput and memory problems often occur. These difficulties increase when utilizing PFA to process low-resource languages, as PFA requires large amounts of data, and the data for low-resource languages are often insufficient. Utilizing the current research premise that NMT model performance can be enhanced through various pre-processing and post-processing strategies without changing the model, we applied various decoding strategies to Korean–English NMT, which relies on a low-resource language pair. Through comparative experiments, we proved that translation performance could be enhanced without changes to the model. We experimentally examined how performance changed in response to beam size changes and n-gram blocking, and whether performance was enhanced when a length penalty was applied. The results showed that various decoding strategies enhance the performance and compare well with previous Korean–English NMT approaches. Therefore, the proposed methodology can improve the performance of NMT models, without the use of PFA; this presents a new perspective for improving machine translation performance.

Factored Statistical Machine Translation for German-English

2018 ◽  
Vol 5 (1) ◽  
pp. 37-45
Author(s):  
Darryl Yunus Sulistyan
Keyword(s):  
Machine Translation ◽  
English Language ◽  
Statistical Machine Translation ◽  
New Model ◽  
Language Pair

Machine Translation is a machine that is going to automatically translate given sentences in a language to other particular language. This paper aims to test the effectiveness of a new model of machine translation which is factored machine translation. We compare the performance of the unfactored system as our baseline compared to the factored model in terms of BLEU score. We test the model in German-English language pair using Europarl corpus. The tools we are using is called MOSES. It is freely downloadable and use. We found, however, that the unfactored model scored over 24 in BLEU and outperforms the factored model which scored below 24 in BLEU for all cases. In terms of words being translated, however, all of factored models outperforms the unfactored model.

Improve Statistical Machine Translation with Context-Sensitive Bilingual Semantic Embedding Model

2014 ◽  
Author(s):  
Haiyang Wu ◽  
Daxiang Dong ◽  
Xiaoguang Hu ◽  
Dianhai Yu ◽  
Wei He ◽  
...  
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Context Sensitive ◽  
Semantic Embedding

Synchronous Tree Sequence Substitution Grammar for Statistical Machine Translation

2009 ◽  
Vol 35 (10) ◽  
pp. 1317-1326
Author(s):  
Hong-Fei JIANG ◽  
Sheng LI ◽  
Min ZHANG ◽  
Tie-Jun ZHAO ◽  
Mu-Yun YANG
Keyword(s):  
Machine Translation ◽  
Statistical Machine Translation ◽  
Sequence Substitution

Analysis Accuracy of Similar Word Based Clustering (EWSB) Algorithm on Machine Translator Bahasa Indonesia-Minang

2018 ◽  
Vol 3 (3) ◽  
Author(s):  
Herry Sujaini
Keyword(s):  
Machine Translation ◽  
Clustering Algorithm ◽  
Statistical Machine Translation ◽  
Target Language ◽  
Word Similarity ◽  
Similar Word ◽  
Word Clustering ◽  
Translation Accuracy ◽  
Bahasa Indonesia

Extended Word Similarity Based (EWSB) Clustering is a word clustering algorithm based on the value of words similarity obtained from the computation of a corpus. One of the benefits of clustering with this algorithm is to improve the translation of a statistical machine translation. Previous research proved that EWSB algorithm could improve the Indonesian-English translator, where the algorithm was applied to Indonesian language as target language.This paper discusses the results of a research using EWSB algorithm on a Indonesian to Minang statistical machine translator, where the algorithm is applied to Minang language as the target language. The research obtained resulted that the EWSB algorithm is quite effective when used in Minang language as the target language. The results of this study indicate that EWSB algorithm can improve the translation accuracy by 6.36%.

Sign in / Sign up

Export Citation Format

Share Document