Text Genre Detection Using Doc2Vec Word-embedding Language Model

2019 ◽  
Vol 23 (2) ◽  
pp. 23-43
Author(s):  
Dongsung Kim
Keyword(s):  
Language Model ◽  
Word Embedding ◽  
Text Genre

scholarly journals A Latent Variable Model for Learning Distributional Relation Vectors

2019 ◽  
Author(s):  
Jose Camacho-Collados ◽  
Luis Espinosa-Anke ◽  
Shoaib Jameel ◽  
Steven Schockaert
Keyword(s):  
Latent Variable ◽  
Language Model ◽  
Word Embedding ◽  
Latent Variable Model ◽  
Variable Model ◽  
The Given ◽  
The Relationship ◽  
Unsupervised Approaches ◽  
Target Words

Recently a number of unsupervised approaches have been proposed for learning vectors that capture the relationship between two words. Inspired by word embedding models, these approaches rely on co-occurrence statistics that are obtained from sentences in which the two target words appear. However, the number of such sentences is often quite small, and most of the words that occur in them are not relevant for characterizing the considered relationship. As a result, standard co-occurrence statistics typically lead to noisy relation vectors. To address this issue, we propose a latent variable model that aims to explicitly determine what words from the given sentences best characterize the relationship between the two target words. Relation vectors then correspond to the parameters of a simple unigram language model which is estimated from these words.

scholarly journals Fracture Mechanics Method for Word Embedding Generation of Neural Probabilistic Linguistic Model

2016 ◽  
Vol 2016 ◽  
pp. 1-11
Author(s):  
Size Bi ◽  
Xiao Liang ◽  
Ting-lei Huang
Keyword(s):  
Fracture Mechanics ◽  
Language Model ◽  
Named Entity Recognition ◽  
Word Embedding ◽  
Entity Recognition ◽  
Semantic Role Labeling ◽  
Part Of Speech ◽  
Linguistic Model ◽  
Traditional Language ◽  
Long Time

Word embedding, a lexical vector representation generated via the neural linguistic model (NLM), is empirically demonstrated to be appropriate for improvement of the performance of traditional language model. However, the supreme dimensionality that is inherent in NLM contributes to the problems of hyperparameters and long-time training in modeling. Here, we propose a force-directed method to improve such problems for simplifying the generation of word embedding. In this framework, each word is assumed as a point in the real world; thus it can approximately simulate the physical movement following certain mechanics. To simulate the variation of meaning in phrases, we use the fracture mechanics to do the formation and breakdown of meaning combined by a 2-gram word group. With the experiments on the natural linguistic tasks of part-of-speech tagging, named entity recognition and semantic role labeling, the result demonstrated that the 2-dimensional word embedding can rival the word embeddings generated by classic NLMs, in terms of accuracy, recall, and text visualization.

scholarly journals G2Basy: A framework to improve the RNN language model and ease overfitting problem

PLoS ONE ◽  
2021 ◽  
Vol 16 (4) ◽  
pp. e0249820
Author(s):  
Lu Yuwen ◽  
Shuyu Chen ◽  
Xiaohan Yuan
Keyword(s):  
Language Model ◽  
Word Embedding ◽  
Language Models ◽  
Batch Size ◽  
Training Process ◽  
Improve Performance ◽  
Step Size ◽  
Learning Rates ◽  
Speed Up ◽  
Overfitting Problem

Recurrent neural networks are efficient ways of training language models, and various RNN networks have been proposed to improve performance. However, with the increase of network scales, the overfitting problem becomes more urgent. In this paper, we propose a framework—G2Basy—to speed up the training process and ease the overfitting problem. Instead of using predefined hyperparameters, we devise a gradient increasing and decreasing technique that changes the parameters training batch size and input dropout simultaneously by a user-defined step size. Together with a pretrained word embedding initialization procedure and the introduction of different optimizers at different learning rates, our framework speeds up the training process dramatically and improves performance compared with a benchmark model of the same scale. For the word embedding initialization, we propose the concept of “artificial features” to describe the characteristics of the obtained word embeddings. We experiment on two of the most often used corpora—the Penn Treebank and WikiText-2 datasets—and both outperform the benchmark results and show potential towards further improvement. Furthermore, our framework shows better results with the larger and more complicated WikiText-2 corpus than with the Penn Treebank. Compared with other state-of-the-art results, we achieve comparable results with network scales hundreds of times smaller and within fewer training epochs.

scholarly journals MenuNER: Domain-Adapted BERT Based NER Approach for a Domain with Limited Dataset and Its Application to Food Menu Domain

2021 ◽  
Vol 11 (13) ◽  
pp. 6007
Author(s):  
Muzamil Hussain Syed ◽  
Sun-Tae Chung
Keyword(s):  
Domain Adaptation ◽  
Language Model ◽  
Named Entity Recognition ◽  
Word Embedding ◽  
Fine Tuning ◽  
Entity Recognition ◽  
Language Models ◽  
Feature Vectors ◽  
Named Entity ◽  
Domain Specific

Entity-based information extraction is one of the main applications of Natural Language Processing (NLP). Recently, deep transfer-learning utilizing contextualized word embedding from pre-trained language models has shown remarkable results for many NLP tasks, including Named-entity recognition (NER). BERT (Bidirectional Encoder Representations from Transformers) is gaining prominent attention among various contextualized word embedding models as a state-of-the-art pre-trained language model. It is quite expensive to train a BERT model from scratch for a new application domain since it needs a huge dataset and enormous computing time. In this paper, we focus on menu entity extraction from online user reviews for the restaurant and propose a simple but effective approach for NER task on a new domain where a large dataset is rarely available or difficult to prepare, such as food menu domain, based on domain adaptation technique for word embedding and fine-tuning the popular NER task network model ‘Bi-LSTM+CRF’ with extended feature vectors. The proposed NER approach (named as ‘MenuNER’) consists of two step-processes: (1) Domain adaptation for target domain; further pre-training of the off-the-shelf BERT language model (BERT-base) in semi-supervised fashion on a domain-specific dataset, and (2) Supervised fine-tuning the popular Bi-LSTM+CRF network for downstream task with extended feature vectors obtained by concatenating word embedding from the domain-adapted pre-trained BERT model from the first step, character embedding and POS tag feature information. Experimental results on handcrafted food menu corpus from customers’ review dataset show that our proposed approach for domain-specific NER task, that is: food menu named-entity recognition, performs significantly better than the one based on the baseline off-the-shelf BERT-base model. The proposed approach achieves 92.5% F1 score on the YELP dataset for the MenuNER task.

scholarly journals Character n-Gram Embeddings to Improve RNN Language Models

2019 ◽  
Vol 33 ◽  
pp. 5074-5082 ◽  
Author(s):  
Sho Takase ◽  
Jun Suzuki ◽  
Masaaki Nagata
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Recurrent Neural Network ◽  
Language Model ◽  
Language Modeling ◽  
Word Embedding ◽  
Experimental Results ◽  
Language Models ◽  
Word Embeddings ◽  
N Gram

This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character ngram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks

Sign in / Sign up

Export Citation Format

Share Document