Understanding Word Embeddings and Language Models

2020 ◽  
pp. 17-31
Author(s):  
Jose Manuel Gomez-Perez ◽  
Ronald Denaux ◽  
Andres Garcia-Silva
Keyword(s):  
Language Models ◽  
Word Embeddings

scholarly journals Better Word Representation Vectors Using Syllabic Alphabet: A Case Study of Swahili

2019 ◽  
Vol 9 (18) ◽  
pp. 3648
Author(s):  
Casper S. Shikali ◽  
Zhou Sijie ◽  
Liu Qihe ◽  
Refuoe Mokhosi
Keyword(s):  
Language Processing ◽  
Critical Role ◽  
Language Model ◽  
Central Africa ◽  
Spoken Language ◽  
Language Models ◽  
Word Embeddings ◽  
Word Representation

Deep learning has extensively been used in natural language processing with sub-word representation vectors playing a critical role. However, this cannot be said of Swahili, which is a low resource and widely spoken language in East and Central Africa. This study proposed novel word embeddings from syllable embeddings (WEFSE) for Swahili to address the concern of word representation for agglutinative and syllabic-based languages. Inspired by the learning methodology of Swahili in beginner classes, we encoded respective syllables instead of characters, character n-grams or morphemes of words and generated quality word embeddings using a convolutional neural network. The quality of WEFSE was demonstrated by the state-of-art results in the syllable-aware language model on both the small dataset (31.229 perplexity value) and the medium dataset (45.859 perplexity value), outperforming character-aware language models. We further evaluated the word embeddings using word analogy task. To the best of our knowledge, syllabic alphabets have not been used to compose the word representation vectors. Therefore, the main contributions of the study are a syllabic alphabet, WEFSE, a syllabic-aware language model and a word analogy dataset for Swahili.

scholarly journals An Introduction to Word Embeddings and Language Models

2021 ◽  
Author(s):  
Tammie Borders ◽  
Svitlana Volkova
Keyword(s):  
Language Models ◽  
Word Embeddings

scholarly journals Inducing Relational Knowledge from BERT

2020 ◽  
Vol 34 (05) ◽  
pp. 7456-7463 ◽  
Author(s):  
Zied Bouraoui ◽  
Jose Camacho-Collados ◽  
Steven Schockaert
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Language Model ◽  
Language Models ◽  
Word Embeddings ◽  
Relational Knowledge ◽  
Wide Range ◽  
Fine Tune ◽  
Standard Word

One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.

Portuguese Language Models and Word Embeddings: Evaluating on Semantic Similarity Tasks

2020 ◽  
pp. 239-248 ◽  
Author(s):  
Ruan Chaves Rodrigues ◽  
Jéssica Rodrigues ◽  
Pedro Vitor Quinta de Castro ◽  
Nádia Felix Felipe da Silva ◽  
Anderson Soares
Keyword(s):  
Semantic Similarity ◽  
Language Models ◽  
Word Embeddings

scholarly journals Using Twitter Data to Monitor Natural Disaster Social Dynamics: A Recurrent Neural Network Approach with Word Embeddings and Kernel Density Estimation

Sensors ◽  
2019 ◽  
Vol 19 (7) ◽  
pp. 1746 ◽  
Author(s):  
Aldo Hernandez-Suarez ◽  
Gabriel Sanchez-Perez ◽  
Karina Toscano-Medina ◽  
Hector Perez-Meana ◽  
Jose Portillo-Portillo ◽  
...  
Keyword(s):  
Neural Network ◽  
Natural Disasters ◽  
Natural Disaster ◽  
Density Estimation ◽  
Recurrent Neural Network ◽  
Kernel Density Estimation ◽  
Kernel Density ◽  
Entity Recognition ◽  
Language Models ◽  
Word Embeddings

In recent years, Online Social Networks (OSNs) have received a great deal of attention for their potential use in the spatial and temporal modeling of events owing to the information that can be extracted from these platforms. Within this context, one of the most latent applications is the monitoring of natural disasters. Vital information posted by OSN users can contribute to relief efforts during and after a catastrophe. Although it is possible to retrieve data from OSNs using embedded geographic information provided by GPS systems, this feature is disabled by default in most cases. An alternative solution is to geoparse specific locations using language models based on Named Entity Recognition (NER) techniques. In this work, a sensor that uses Twitter is proposed to monitor natural disasters. The approach is intended to sense data by detecting toponyms (named places written within the text) in tweets with event-related information, e.g., a collapsed building on a specific avenue or the location at which a person was last seen. The proposed approach is carried out by transforming tokenized tweets into word embeddings: a rich linguistic and contextual vector representation of textual corpora. Pre-labeled word embeddings are employed to train a Recurrent Neural Network variant, known as a Bidirectional Long Short-Term Memory (biLSTM) network, that is capable of dealing with sequential data by analyzing information in both directions of a word (past and future entries). Moreover, a Conditional Random Field (CRF) output layer, which aims to maximize the transition from one NER tag to another, is used to increase the classification accuracy. The resulting labeled words are joined to coherently form a toponym, which is geocoded and scored by a Kernel Density Estimation function. At the end of the process, the scored data are presented graphically to depict areas in which the majority of tweets reporting topics related to a natural disaster are concentrated. A case study on Mexico’s 2017 Earthquake is presented, and the data extracted during and after the event are reported.

scholarly journals Comparing the Quality and Speed of Sentence Classification with Modern Language Models

2020 ◽  
Vol 10 (10) ◽  
pp. 3386 ◽  
Author(s):  
Krzysztof Fiok ◽  
Waldemar Karwowski ◽  
Edgar Gutierrez ◽  
Mohammad Reza-Davahli
Keyword(s):  
Language Models ◽  
Data Sets ◽  
Word Embeddings ◽  
Data Set ◽  
Sentence Level ◽  
Sentence Classification ◽  
Development Of Language ◽  
Gated Recurrent Units ◽  
Vector Representations ◽  
Fully Connected

After the advent of Glove and Word2vec, the dynamic development of language models (LMs) used to generate word embeddings has enabled the creation of better text classifier frameworks. With the vector representations of words generated by newer LMs, embeddings are no longer static but are context-aware. However, the quality of results provided by state-of-the-art LMs comes at the price of speed. Our goal was to present a benchmark to provide insight into the speed–quality trade-off of a sentence classifier framework based on word embeddings provided by selected LMs. We used a recurrent neural network with gated recurrent units to create sentence-level vector representations from word embeddings provided by an LM and a single fully connected layer for classification. Benchmarking was performed on two sentence classification data sets: The Sixth Text REtrieval Conference (TREC6)set and a 1000-sentence data set of our design. Our Monte Carlo cross-validated results based on these two data sources demonstrated that the newest deep learning LMs provided improvements over Glove and FastText in terms of weighted Matthews correlation coefficient (MCC) scores. We postulate that progress in LMs is more apparent when more difficult classification tasks are addressed.

scholarly journals Character n-Gram Embeddings to Improve RNN Language Models

2019 ◽  
Vol 33 ◽  
pp. 5074-5082 ◽  
Author(s):  
Sho Takase ◽  
Jun Suzuki ◽  
Masaaki Nagata
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Recurrent Neural Network ◽  
Language Model ◽  
Language Modeling ◽  
Word Embedding ◽  
Experimental Results ◽  
Language Models ◽  
Word Embeddings ◽  
N Gram

This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character ngram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks

Dissecting word embeddings and language models in natural language processing

2021 ◽  
Vol 24 (5) ◽  
pp. 1509-1515
Author(s):  
Vivek Kumar Verma ◽  
Mrigank Pandey ◽  
Tarun Jain ◽  
Pradeep Kumar Tiwari
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Language Models ◽  
Word Embeddings
Sign in / Sign up

Export Citation Format

Share Document