Enhanced Text Stemmer with Noisy Text Normalization for Malay Texts

2019 ◽  
pp. 433-444
Author(s):  
Mohamad Nizam Kassim ◽  
Shaiful Hisham Mat Jali ◽  
Mohd Aizaini Maarof ◽  
Anazida Zainal ◽  
Amirudin Abdul Wahab
Keyword(s):  
Noisy Text ◽  
Text Normalization

Normalization of Transliterated Mongolian Words Using Seq2Seq Model with Limited Data

2021 ◽  
Vol 20 (6) ◽  
pp. 1-19
Author(s):  
Zolzaya Byambadorj ◽  
Ryota Nishimura ◽  
Altangerel Ayush ◽  
Norihide Kitaoka
Keyword(s):  
Social Media ◽  
Performance Enhancement ◽  
Statistical Machine Translation ◽  
Training Data ◽  
Daily Lives ◽  
Latin Script ◽  
N Gram ◽  
Noisy Text ◽  
Different Cultures ◽  
Text Normalization

The huge increase in social media use in recent years has resulted in new forms of social interaction, changing our daily lives. Due to increasing contact between people from different cultures as a result of globalization, there has also been an increase in the use of the Latin alphabet, and as a result a large amount of transliterated text is being used on social media. In this study, we propose a variety of character level sequence-to-sequence (seq2seq) models for normalizing noisy, transliterated text written in Latin script into Mongolian Cyrillic script, for scenarios in which there is a limited amount of training data available. We applied performance enhancement methods, which included various beam search strategies, N-gram-based context adoption, edit distance-based correction and dictionary-based checking, in novel ways to two basic seq2seq models. We experimentally evaluated these two basic models as well as fourteen enhanced seq2seq models, and compared their noisy text normalization performance with that of a transliteration model and a conventional statistical machine translation (SMT) model. The proposed seq2seq models improved the robustness of the basic seq2seq models for normalizing out-of-vocabulary (OOV) words, and most of our models achieved higher normalization performance than the conventional method. When using test data during our text normalization experiment, our proposed method which included checking each hypothesis during the inference period achieved the lowest word error rate (WER = 13.41%), which was 4.51% fewer errors than when using the conventional SMT method.

Low-resource neural character-based noisy text normalization

2019 ◽  
Vol 36 (5) ◽  
pp. 4921-4929 ◽  
Author(s):  
Manuel Mager ◽  
Mónica Jasso Rosales ◽  
Özlem Çetinoğlu ◽  
Ivan Meza
Keyword(s):  
Low Resource ◽  
Noisy Text ◽  
Text Normalization

scholarly journals An Unsupervised Normalization Algorithm for Noisy Text: A Case Study for Information Retrieval and Stance Detection

2021 ◽  
Vol 13 (3) ◽  
pp. 1-25
Author(s):  
Anurag Roy ◽  
Shalmoli Ghosh ◽  
Kripabandhu Ghosh ◽  
Saptarshi Ghosh
Keyword(s):  
Large Fraction ◽  
Training Data ◽  
Writing Style ◽  
Robust Algorithms ◽  
Normalization Methods ◽  
Textual Data ◽  
Digitized Documents ◽  
Noisy Text ◽  
Text Normalization

A large fraction of textual data available today contains various types of “noise,” such as OCR noise in digitized documents, noise due to informal writing style of users on microblogging sites, and so on. To enable tasks such as search/retrieval and classification over all the available data, we need robust algorithms for text normalization, i.e., for cleaning different kinds of noise in the text. There have been several efforts towards cleaning or normalizing noisy text; however, many of the existing text normalization methods are supervised and require language-dependent resources or large amounts of training data that is difficult to obtain. We propose an unsupervised algorithm for text normalization that does not need any training data/human intervention. The proposed algorithm is applicable to text over different languages and can handle both machine-generated and human-generated noise. Experiments over several standard datasets show that text normalization through the proposed algorithm enables better retrieval and stance detection, as compared to that using several baseline text normalization methods.

Improving Text Normalization via Unsupervised Model and Discriminative Reranking

2014 ◽  
Author(s):  
Chen Li ◽  
Yang Liu
Keyword(s):  
Text Normalization

A Mostly Data-Driven Approach to Inverse Text Normalization

2017 ◽  
Author(s):  
Ernest Pusateri ◽  
Bharat Ram Ambati ◽  
Elizabeth Brooks ◽  
Ondrej Platek ◽  
Donald McAllaster ◽  
...  
Keyword(s):  
Data Driven ◽  
Data Driven Approach ◽  
Text Normalization

Efficient Text Normalization via Hybrid Bi-directional LSTM

2021 ◽  
Author(s):  
Divya ◽  
Sunil Kumar ◽  
Debanjan Sadhya ◽  
Santosh Singh Rathore
Keyword(s):  
Text Normalization

scholarly journals NeMo Inverse Text Normalization: From Development to Production

2021 ◽  
Author(s):  
Yang Zhang ◽  
Evelina Bakhturina ◽  
Kyle Gorman ◽  
Boris Ginsburg
Keyword(s):  
Text Normalization

PANDA - Discovering Part Name in Noisy Text Data

2018 ◽  
Author(s):  
Anne Kao ◽  
Nobal B Niraula ◽  
Daniel Whyatt
Keyword(s):  
Text Data ◽  
Noisy Text
Sign in / Sign up

Export Citation Format

Share Document