scholarly journals Near-Lossless Binarization of Word Embeddings

2019 ◽  
Vol 33 ◽  
pp. 7104-7111 ◽  
Author(s):  
Julien Tissier ◽  
Christophe Gravier ◽  
Amaury Habrard
Keyword(s):  
Sentiment Analysis ◽  
Semantic Similarity ◽  
Text Classification ◽  
Semantic Information ◽  
State Of The Art ◽  
Floating Point ◽  
Word Embeddings ◽  
Binary Vectors ◽  
Starting Point ◽  
Memory Footprint

Word embeddings are commonly used as a starting point in many NLP models to achieve state-of-the-art performances. However, with a large vocabulary and many dimensions, these floating-point representations are expensive both in terms of memory and calculations which makes them unsuitable for use on low-resource devices. The method proposed in this paper transforms real-valued embeddings into binary embeddings while preserving semantic information, requiring only 128 or 256 bits for each vector. This leads to a small memory footprint and fast vector operations. The model is based on an autoencoder architecture, which also allows to reconstruct original vectors from the binary ones. Experimental results on semantic similarity, text classification and sentiment analysis tasks show that the binarization of word embeddings only leads to a loss of ∼2% in accuracy while vector size is reduced by 97%. Furthermore, a top-k benchmark demonstrates that using these binary vectors is 30 times faster than using real-valued vectors.

scholarly journals Pronunciation-Enhanced Chinese Word Embedding

2021 ◽  
Author(s):  
Qinjuan Yang ◽  
Haoran Xie ◽  
Gary Cheng ◽  
Fu Lee Wang ◽  
Yanghui Rao
Keyword(s):  
Sentiment Analysis ◽  
Text Classification ◽  
Semantic Information ◽  
Word Embedding ◽  
Chinese Characters ◽  
Learning Method ◽  
Word Embeddings ◽  
Chinese Word ◽  
Word Similarity ◽  
Meaning Structure

AbstractChinese word embeddings have recently garnered considerable attention. Chinese characters and their sub-character components, which contain rich semantic information, are incorporated to learn Chinese word embeddings. Chinese characters can represent a combination of meaning, structure, and pronunciation. However, existing embedding learning methods focus on the structure and meaning of Chinese characters. In this study, we aim to develop an embedding learning method that can make complete use of the information represented by Chinese characters, including phonology, morphology, and semantics. Specifically, we propose a pronunciation-enhanced Chinese word embedding learning method, where the pronunciations of context characters and target characters are simultaneously encoded into the embeddings. Evaluation of word similarity, word analogy reasoning, text classification, and sentiment analysis validate the effectiveness of our proposed method.

Deep Learning for text in limted data settings

2020 ◽  
Author(s):  
Pathikkumar Patel ◽  
Bhargav Lad ◽  
Jinan Fiaidhi
Keyword(s):  
Machine Learning ◽  
Time Series ◽  
Deep Learning ◽  
Sentiment Analysis ◽  
Transfer Learning ◽  
Text Classification ◽  
State Of The Art ◽  
Time Series Forecasting ◽  
Text Data ◽  
Performance Levels

During the last few years, RNN models have been extensively used and they have proven to be better for sequence and text data. RNNs have achieved state-of-the-art performance levels in several applications such as text classification, sequence to sequence modelling and time series forecasting. In this article we will review different Machine Learning and Deep Learning based approaches for text data and look at the results obtained from these methods. This work also explores the use of transfer learning in NLP and how it affects the performance of models on a specific application of sentiment analysis.

Learning emotional word embeddings for sentiment analysis

2021 ◽  
pp. 1-13
Author(s):  
Qingtian Zeng ◽  
Xishi Zhao ◽  
Xiaohui Hu ◽  
Hua Duan ◽  
Zhongying Zhao ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Emotional Word ◽  
Classification Model ◽  
Data Sets ◽  
Word Embeddings ◽  
Real World Data ◽  
Text Documents

Word embeddings have been successfully applied in many natural language processing tasks due to its their effectiveness. However, the state-of-the-art algorithms for learning word representations from large amounts of text documents ignore emotional information, which is a significant research problem that must be addressed. To solve the above problem, we propose an emotional word embedding (EWE) model for sentiment analysis in this paper. This method first applies pre-trained word vectors to represent document features using two different linear weighting methods. Then, the resulting document vectors are input to a classification model and used to train a text sentiment classifier, which is based on a neural network. In this way, the emotional polarity of the text is propagated into the word vectors. The experimental results on three kinds of real-world data sets demonstrate that the proposed EWE model achieves superior performances on text sentiment prediction, text similarity calculation, and word emotional expression tasks compared to other state-of-the-art models.

scholarly journals Specializing Word Embeddings (for Parsing) by Information Bottleneck (Extended Abstract)

2020 ◽  
Author(s):  
Xiang Lisa Li ◽  
Jason Eisner
Keyword(s):  
Dimensionality Reduction ◽  
Semantic Information ◽  
State Of The Art ◽  
Word Embedding ◽  
Discrete Version ◽  
Word Embeddings ◽  
Continuous Version ◽  
Continuous Vector ◽  
Information Bottleneck ◽  
Art Performance

Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.

scholarly journals AutoExtend: Combining Word Embeddings with Semantic Resources

2017 ◽  
Vol 43 (3) ◽  
pp. 593-617 ◽  
Author(s):  
Sascha Rothe ◽  
Hinrich Schütze
Keyword(s):  
Semantic Information ◽  
State Of The Art ◽  
Word Sense Disambiguation ◽  
Input Word ◽  
Word Sense ◽  
Word Embeddings ◽  
Training Corpus ◽  
Context Similarity ◽  
Sense Disambiguation ◽  
Semantic Resources

We present AutoExtend, a system that combines word embeddings with semantic resources by learning embeddings for non-word objects like synsets and entities and learning word embeddings that incorporate the semantic information from the resource. The method is based on encoding and decoding the word embeddings and is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The obtained embeddings live in the same vector space as the input word embeddings. A sparse tensor formalization guarantees efficiency and parallelizability. We use WordNet, GermaNet, and Freebase as semantic resources. AutoExtend achieves state-of-the-art performance on Word-in-Context Similarity and Word Sense Disambiguation tasks.

scholarly journals Text classification with semantically enriched word embeddings

2020 ◽  
pp. 1-35
Author(s):  
N. Pittaras ◽  
G. Giannakopoulos ◽  
G. Papadakis ◽  
V. Karkaletsis
Keyword(s):  
Text Classification ◽  
Semantic Information ◽  
Classification Performance ◽  
Classification Task ◽  
Propagation Mechanism ◽  
Word Embeddings ◽  
Performance Loss ◽  
Part Of Speech ◽  
Box Models ◽  
Document Frequency

Abstract The recent breakthroughs in deep neural architectures across multiple machine learning fields have led to the widespread use of deep neural models. These learners are often applied as black-box models that ignore or insufficiently utilize a wealth of preexisting semantic information. In this study, we focus on the text classification task, investigating methods for augmenting the input to deep neural networks (DNNs) with semantic information. We extract semantics for the words in the preprocessed text from the WordNet semantic graph, in the form of weighted concept terms that form a semantic frequency vector. Concepts are selected via a variety of semantic disambiguation techniques, including a basic, a part-of-speech-based, and a semantic embedding projection method. Additionally, we consider a weight propagation mechanism that exploits semantic relationships in the concept graph and conveys a spreading activation component. We enrich word2vec embeddings with the resulting semantic vector through concatenation or replacement and apply the semantically augmented word embeddings on the classification task via a DNN. Experimental results over established datasets demonstrate that our approach of semantic augmentation in the input space boosts classification performance significantly, with concatenation offering the best performance. We also note additional interesting findings produced by our approach regarding the behavior of term frequency - inverse document frequency normalization on semantic vectors, along with the radical dimensionality reduction potential with negligible performance loss.

scholarly journals An Empirical Study of Korean Sentence Representation with Various Tokenizations

Electronics ◽  
2021 ◽  
Vol 10 (7) ◽  
pp. 845
Author(s):  
Danbi Cho ◽  
Hyunyoung Lee ◽  
Seungshik Kang
Keyword(s):  
Empirical Study ◽  
Natural Language ◽  
Sentiment Analysis ◽  
Machine Translation ◽  
Text Classification ◽  
State Of The Art ◽  
Language Models ◽  
Vocabulary Size ◽  
Analysis Task ◽  
Natural Language Process

It is important how the token unit is defined in a sentence in natural language process tasks, such as text classification, machine translation, and generation. Many studies recently utilized the subword tokenization in language models such as BERT, KoBERT, and ALBERT. Although these language models achieved state-of-the-art results in various NLP tasks, it is not clear whether the subword tokenization is the best token unit for Korean sentence embedding. Thus, we carried out sentence embedding based on word, morpheme, subword, and submorpheme, respectively, on Korean sentiment analysis. We explored the two-sentence representation methods for sentence embedding: considering the order of tokens in a sentence and not considering the order. While inputting a sentence, which is decomposed by token unit, to the two-sentence representation methods, we construct the sentence embedding with various tokenizations to find the most effective token unit for Korean sentence embedding. In our work, we confirmed: the robustness of the subword unit for out-of-vocabulary (OOV) problems compared to other token units, the disadvantage of replacing whitespace with a particular symbol in the sentiment analysis task, and that the optimal vocabulary size is 16K in subword and submorpheme tokenization. We empirically noticed that the subword, which was tokenized by a vocabulary size of 16K without replacement of whitespace, was the most effective for sentence embedding on the Korean sentiment analysis task.

Sentiment analysis in Twitter

2012 ◽  
Vol 20 (1) ◽  
pp. 1-28 ◽  
Author(s):  
EUGENIO MARTÍNEZ-CÁMARA ◽  
M. TERESA MARTÍN-VALDIVIA ◽  
L. ALFONSO UREÑA-LÓPEZ ◽  
A RTURO MONTEJO-RÁEZ
Keyword(s):  
Sentiment Analysis ◽  
Recommender Systems ◽  
State Of The Art ◽  
Research Community ◽  
The Other ◽  
Scientific Publications ◽  
Graphical Interfaces ◽  
Starting Point ◽  
Potential Applications ◽  
Virtual Assistance

AbstractIn recent years, the interest among the research community in sentiment analysis (SA) has grown exponentially. It is only necessary to see the number of scientific publications and forums or related conferences to understand that this is a field with great prospects for the future. On the other hand, the Twitter boom has boosted investigation in this area due fundamentally to its potential applications in areas such as business or government intelligence, recommender systems, graphical interfaces and virtual assistance. However, to fully understand this issue, a profound revision of the state of the art is first necessary. It is for this reason that this paper aims to represent a starting point for those investigations concerned with the latest references to Twitter in SA.

scholarly journals Causally Denoise Word Embeddings Using Half-Sibling Regression

2020 ◽  
Vol 34 (05) ◽  
pp. 9426-9433 ◽  
Author(s):  
Zekun Yang ◽  
Tianlin Liu
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Causal Inference ◽  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Word Embeddings ◽  
Modern Natural ◽  
Growing Body ◽  
Half Sibling

Distributional representations of words, also known as word vectors, have become crucial for modern natural language processing tasks due to their wide applications. Recently, a growing body of word vector postprocessing algorithm has emerged, aiming to render off-the-shelf word vectors even stronger. In line with these investigations, we introduce a novel word vector postprocessing scheme under a causal inference framework. Concretely, the postprocessing pipeline is realized by Half-Sibling Regression (HSR), which allows us to identify and remove confounding noise contained in word vectors. Compared to previous work, our proposed method has the advantages of interpretability and transparency due to its causal inference grounding. Evaluated on a battery of standard lexical-level evaluation tasks and downstream sentiment analysis tasks, our method reaches state-of-the-art performance.

Joint Embedding of Words and Labels for Sentiment Classification

2021 ◽  
Vol 31 (01) ◽  
Author(s):  
Yingwei Sheng ◽  
Inui Takashi
Keyword(s):  
Social Networks ◽  
Sentiment Analysis ◽  
Text Classification ◽  
Research Topic ◽  
Sentiment Classification ◽  
Word Embeddings ◽  
Baseline Model ◽  
Traditional Methods ◽  
Joint Embedding ◽  
The Web

With the fast growth of social networks, sentiment analysis on the web has been a popular research topic. Recently, word embedding-based sentiment analysis methods have reached outstanding performance compared to the traditional methods. However, word embeddings always ignore information from dataset’s labels. Inspired by the LEAM model proposed by Wang et al. [Joint embedding of words and labels for text classification (2018), arXiv:1805.04174], we propose a method that jointly learns information of words and sentiment labels, which can improve the performance of the label embedding model. We defined a set of sentiment lexicons and used it to represent sentiment labels in the proposed method. We finally conducted experiments on the Yelp dataset, which reached 65.03% accuracy when using the same setup as the baseline model, and 65.22% accuracy when using optional window sizes.

Sign in / Sign up

Export Citation Format

Share Document