An Adaptive Wordpiece Language Model for Learning Chinese Word Embeddings

2019 ◽  
Author(s):  
BinChen Xu ◽  
Lu Ma ◽  
Liang Zhang ◽  
HaoHai Li ◽  
Qi Kang ◽  
...  
Keyword(s):  
Language Model ◽  
Word Embeddings ◽  
Chinese Word

scholarly journals Joint Fine-Grained Components Continuously Enhance Chinese Word Embeddings

IEEE Access ◽  
2019 ◽  
Vol 7 ◽  
pp. 174699-174708
Author(s):  
Chengyang Zhuang ◽  
Yuanjie Zheng ◽  
Wenhui Huang ◽  
Weikuan Jia
Keyword(s):  
Word Embeddings ◽  
Chinese Word ◽  
Fine Grained

scholarly journals Word embeddings for application in geosciences: development, evaluation and examples of soil-related concepts

2019 ◽  
Author(s):  
José Padarian ◽  
Ignacio Fuentes
Keyword(s):  
Language Processing ◽  
Dimensional Space ◽  
Language Model ◽  
Test Suite ◽  
Word Embeddings ◽  
General Domain ◽  
Domain Specific ◽  
Descriptive Information ◽  
Development Evaluation ◽  
Numerical Representations

Abstract. A large amount of descriptive information is available in most disciplines of geosciences. This information is usually considered subjective and ill-favoured compared with its numerical counterpart. Considering the advances in natural language processing and machine learning, it is possible to utilise descriptive information and encode it as dense vectors. These word embeddings lay on a multi-dimensional space where angles and distances have a linguistic interpretation. We used 280 764 full-text scientific articles related to geosciences to train a domain-specific language model capable of generating such embeddings. To evaluate the quality of the numerical representations, we performed three intrinsic evaluations, namely: the capacity to generate analogies, term relatedness compared with the opinion of a human subject, and categorisation of different groups of words. Since this is the first attempt to evaluate word embedding for tasks in the geosciences domain, we created a test suite specific for geosciences. We compared our results with general domain embeddings commonly used in other disciplines. As expected, our domain-specific embeddings (GeoVec) outperformed general domain embeddings in all tasks, with an overall performance improvement of 107.9 %. The resulting embedding and test suite will be made available for other researchers to use an expand.

scholarly journals Better Word Representation Vectors Using Syllabic Alphabet: A Case Study of Swahili

2019 ◽  
Vol 9 (18) ◽  
pp. 3648
Author(s):  
Casper S. Shikali ◽  
Zhou Sijie ◽  
Liu Qihe ◽  
Refuoe Mokhosi
Keyword(s):  
Language Processing ◽  
Critical Role ◽  
Language Model ◽  
Central Africa ◽  
Spoken Language ◽  
Language Models ◽  
Word Embeddings ◽  
Word Representation

Deep learning has extensively been used in natural language processing with sub-word representation vectors playing a critical role. However, this cannot be said of Swahili, which is a low resource and widely spoken language in East and Central Africa. This study proposed novel word embeddings from syllable embeddings (WEFSE) for Swahili to address the concern of word representation for agglutinative and syllabic-based languages. Inspired by the learning methodology of Swahili in beginner classes, we encoded respective syllables instead of characters, character n-grams or morphemes of words and generated quality word embeddings using a convolutional neural network. The quality of WEFSE was demonstrated by the state-of-art results in the syllable-aware language model on both the small dataset (31.229 perplexity value) and the medium dataset (45.859 perplexity value), outperforming character-aware language models. We further evaluated the word embeddings using word analogy task. To the best of our knowledge, syllabic alphabets have not been used to compose the word representation vectors. Therefore, the main contributions of the study are a syllabic alphabet, WEFSE, a syllabic-aware language model and a word analogy dataset for Swahili.

scholarly journals Learning Chinese Word Embeddings With Words and Subcharacter N-Grams

IEEE Access ◽  
2019 ◽  
Vol 7 ◽  
pp. 42987-42992 ◽  
Author(s):  
Ruizhi Kang ◽  
Hongjun Zhang ◽  
Wenning Hao ◽  
Kai Cheng ◽  
Guanglu Zhang
Keyword(s):  
Word Embeddings ◽  
Chinese Word

Learning Syllables Using Conv-LSTM Model for Swahili Word Representation and Part-of-speech Tagging

2021 ◽  
Vol 20 (4) ◽  
pp. 1-25
Author(s):  
Casper Shikali Shivachi ◽  
Refuoe Mokhosi ◽  
Zhou Shijie ◽  
Liu Qihe
Keyword(s):  
Language Processing ◽  
Short Term Memory ◽  
Language Model ◽  
Word Embeddings ◽  
Short Term ◽  
Term Memory ◽  
Pos Tagging ◽  
Part Of Speech ◽  
Word Representation ◽  
Long Short Term Memory

The need to capture intra-word information in natural language processing (NLP) tasks has inspired research in learning various word representations at word, character, or morpheme levels, but little attention has been given to syllables from a syllabic alphabet. Motivated by the success of compositional models in morphological languages, we present a Convolutional-long short term memory (Conv-LSTM) model for constructing Swahili word representation vectors from syllables. The unified architecture addresses the word agglutination and polysemous nature of Swahili by extracting high-level syllable features using a convolutional neural network (CNN) and then composes quality word embeddings with a long short term memory (LSTM). The word embeddings are then validated using a syllable-aware language model ( 31.267 ) and a part-of-speech (POS) tagging task ( 98.78 ), both yielding very competitive results to the state-of-art models in their respective domains. We further validate the language model using Xhosa and Shona, which are syllabic-based languages. The novelty of the study is in its capability to construct quality word embeddings from syllables using a hybrid model that does not use max-over-pool common in CNN and then the exploitation of these embeddings in POS tagging. Therefore, the study plays a crucial role in the processing of agglutinative and syllabic-based languages by contributing quality word embeddings from syllable embeddings, a robust Conv–LSTM model that learns syllables for not only language modeling and POS tagging, but also for other downstream NLP tasks.

scholarly journals Pronunciation-Enhanced Chinese Word Embedding

2021 ◽  
Author(s):  
Qinjuan Yang ◽  
Haoran Xie ◽  
Gary Cheng ◽  
Fu Lee Wang ◽  
Yanghui Rao
Keyword(s):  
Sentiment Analysis ◽  
Text Classification ◽  
Semantic Information ◽  
Word Embedding ◽  
Chinese Characters ◽  
Learning Method ◽  
Word Embeddings ◽  
Chinese Word ◽  
Word Similarity ◽  
Meaning Structure

AbstractChinese word embeddings have recently garnered considerable attention. Chinese characters and their sub-character components, which contain rich semantic information, are incorporated to learn Chinese word embeddings. Chinese characters can represent a combination of meaning, structure, and pronunciation. However, existing embedding learning methods focus on the structure and meaning of Chinese characters. In this study, we aim to develop an embedding learning method that can make complete use of the information represented by Chinese characters, including phonology, morphology, and semantics. Specifically, we propose a pronunciation-enhanced Chinese word embedding learning method, where the pronunciations of context characters and target characters are simultaneously encoded into the embeddings. Evaluation of word similarity, word analogy reasoning, text classification, and sentiment analysis validate the effectiveness of our proposed method.

Sign in / Sign up

Export Citation Format

Share Document