scholarly journals Word Embeddings as Metric Recovery in Semantic Spaces

2016 ◽  
Vol 4 ◽  
pp. 273-286 ◽  
Author(s):  
Tatsunori B. Hashimoto ◽  
David Alvarez-Melis ◽  
Tommi S. Jaakkola
Keyword(s):  
Random Walks ◽  
Manifold Learning ◽  
State Of The Art ◽  
Inductive Reasoning ◽  
Semantic Space ◽  
Word Embedding ◽  
Word Embeddings ◽  
Recovery Algorithm ◽  
Series Completion ◽  
Semantic Spaces

Continuous word representations have been remarkably useful across NLP tasks but remain poorly understood. We ground word embeddings in semantic spaces studied in the cognitive-psychometric literature, taking these spaces as the primary objects to recover. To this end, we relate log co-occurrences of words in large corpora to semantic similarity assessments and show that co-occurrences are indeed consistent with an Euclidean semantic space hypothesis. Framing word embedding as metric recovery of a semantic space unifies existing word embedding algorithms, ties them to manifold learning, and demonstrates that existing algorithms are consistent metric recovery methods given co-occurrence counts from random walks. Furthermore, we propose a simple, principled, direct metric recovery algorithm that performs on par with the state-of-the-art word embedding and manifold learning methods. Finally, we complement recent focus on analogies by constructing two new inductive reasoning datasets—series completion and classification—and demonstrate that word embeddings can be used to solve them as well.

scholarly journals Word-embeddings Italian semantic spaces: A semantic model for psycholinguistic research

Psihologija ◽  
2017 ◽  
Vol 50 (4) ◽  
pp. 503-520 ◽  
Author(s):  
Marco Marelli
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Semantic Space ◽  
Distributional Semantics ◽  
Semantic Model ◽  
Word Embeddings ◽  
Psycholinguistic Research ◽  
Graphic Interface ◽  
Test Sets ◽  
Semantic Spaces

Distributional semantics has been for long a source of successful models in psycholinguistics, permitting to obtain semantic estimates for a large number of words in an automatic and fast way. However, resources in this respect remain scarce or limitedly accessible for languages different from English. The present paper describes WEISS (Word-Embeddings Italian Semantic Space), a distributional semantic model based on Italian. WEISS includes models of semantic representations that are trained adopting state-of-the-art word-embeddings methods, applying neural networks to induce distributed representations for lexical meanings. The resource is evaluated against two test sets, demonstrating that WEISS obtains a better performance with respect to a baseline encoding word associations. Moreover, an extensive qualitative analysis of the WEISS output provides examples of the model potentialities in capturing several semantic phenomena. Two variants of WEISS are released and made easily accessible via web through the SNAUT graphic interface.

scholarly journals Specializing Word Embeddings (for Parsing) by Information Bottleneck (Extended Abstract)

2020 ◽  
Author(s):  
Xiang Lisa Li ◽  
Jason Eisner
Keyword(s):  
Dimensionality Reduction ◽  
Semantic Information ◽  
State Of The Art ◽  
Word Embedding ◽  
Discrete Version ◽  
Word Embeddings ◽  
Continuous Version ◽  
Continuous Vector ◽  
Information Bottleneck ◽  
Art Performance

Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.

scholarly journals A Co-Embedding Model with Variational Auto-Encoder for Knowledge Graphs

2022 ◽  
Vol 12 (2) ◽  
pp. 715
Author(s):  
Luodi Xie ◽  
Huimin Huang ◽  
Qing Du
Keyword(s):  
State Of The Art ◽  
Relation Extraction ◽  
Semantic Space ◽  
Knowledge Graph ◽  
High Quality ◽  
Gaussian Distributions ◽  
Benchmark Datasets ◽  
Semantic Spaces ◽  
Knowledge Graphs ◽  
Low Dimensional

Knowledge graph (KG) embedding has been widely studied to obtain low-dimensional representations for entities and relations. It serves as the basis for downstream tasks, such as KG completion and relation extraction. Traditional KG embedding techniques usually represent entities/relations as vectors or tensors, mapping them in different semantic spaces and ignoring the uncertainties. The affinities between entities and relations are ambiguous when they are not embedded in the same latent spaces. In this paper, we incorporate a co-embedding model for KG embedding, which learns low-dimensional representations of both entities and relations in the same semantic space. To address the issue of neglecting uncertainty for KG components, we propose a variational auto-encoder that represents KG components as Gaussian distributions. In addition, compared with previous methods, our method has the advantages of high quality and interpretability. Our experimental results on several benchmark datasets demonstrate our model’s superiority over the state-of-the-art baselines.

scholarly journals A Causal Inference Method for Reducing Gender Bias in Word Embedding Relations

2020 ◽  
Vol 34 (05) ◽  
pp. 9434-9441
Author(s):  
Zekun Yang ◽  
Juan Feng
Keyword(s):  
Gender Bias ◽  
Language Processing ◽  
State Of The Art ◽  
Word Embedding ◽  
Coreference Resolution ◽  
Word Embeddings ◽  
Inference Method ◽  
Sentence Level ◽  
Statistical Dependency ◽  
And Gender

Word embedding has become essential for natural language processing as it boosts empirical performances of various tasks. However, recent research discovers that gender bias is incorporated in neural word embeddings, and downstream tasks that rely on these biased word vectors also produce gender-biased results. While some word-embedding gender-debiasing methods have been developed, these methods mainly focus on reducing gender bias associated with gender direction and fail to reduce the gender bias presented in word embedding relations. In this paper, we design a causal and simple approach for mitigating gender bias in word vector relation by utilizing the statistical dependency between gender-definition word embeddings and gender-biased word embeddings. Our method attains state-of-the-art results on gender-debiasing tasks, lexical- and sentence-level evaluation tasks, and downstream coreference resolution tasks.

scholarly journals Semantische Äquivalenz in Geburtserzählungen: Anwendung von Word Embeddings

2020 ◽  
Vol 48 (3) ◽  
pp. 562-589
Author(s):  
Noah Bubenhofer
Keyword(s):  
Cultural Analysis ◽  
Word Embedding ◽  
Semantic Relations ◽  
Functional Equivalence ◽  
Word Embeddings ◽  
Online Forums ◽  
Cultural Narratives ◽  
Narrative Patterns ◽  
Semantic Spaces ◽  
Birth Story

AbstractThe present study focuses on serially occurring narrations of ‘everyday’ life, more specifically on birthing as narrated by mothers on online forums; the underlying idea being that these narrations happen against the background of cultural narratives.The present paper uses word embedding models to detect typical topics and actors in these narrations. The calculation of word embeddings automatically constructs semantic spaces, where semantic relations (synonymy in particular) can be modeled. This method offers a way to think of synonymy as ‘functional equivalence in discourse’.The present study relies on previous work with n-grams (Bubenhofer, 2018). N-grams are sequences of words that often appear together; their sequential order in different narrations gives insight in narrative patterns. A further step in the analysis is the construction of ‘narrative topoi’, which is achieved through clustering neighboring vectors. The emerging clusters can in turn be grouped into five narrative elements of ‘telling a birth story’: 1) disruption of daily life, 2) personnel, 3) body, 4) fear, 5) joy. While it seems obvious that certain themes ‘belong’ into the narration of a delivery, it is less obvious with what vocabulary these themes are expressed.The presented method of clustering word-embedding-profiles adds tremendously to the modelling of a narrative. Its advantages lie in its potential to show lexical variation, as it also includes rare, non-conformative orthographical variants. Furthermore, it allows for a discourse-specific (and usage-based) view on semantic relations. The same applies to relations between semantic clusters. Seen from a discourse linguistics or cultural analysis perspective, word embeddings renew our understanding of semantics. This shows particularly fruitful if used to analyze (discourse dependent) derivations between semantic spaces.

Zero-Shot Learning for IMU-Based Activity Recognition Using Video Embeddings

2021 ◽  
Vol 5 (4) ◽  
pp. 1-23
Author(s):  
Catherine Tong ◽  
Jinchen Ge ◽  
Nicholas D. Lane
Keyword(s):  
Activity Recognition ◽  
State Of The Art ◽  
Recognition Performance ◽  
Semantic Space ◽  
Test Time ◽  
Related Information ◽  
Desirable Feature ◽  
Semantic Spaces ◽  
Public Datasets ◽  
Attribute Space

The Activity Recognition Chain generally precludes the challenging scenario of recognizing new activities that were unseen during training, despite this scenario being a practical and common one as users perform diverse activities at test time. A few prior works have adopted zero-shot learning methods for IMU-based activity recognition, which work by relating seen and unseen classes through an auxiliary semantic space. However, these methods usually rely heavily on a hand-crafted attribute space which is costly to define, or a learnt semantic space based on word embedding, which lacks motion-related information crucial for distinguishing IMU features. Instead, we propose a strategy to exploit videos of human activities to construct an informative semantic space. With our approach, knowledge from state-of-the-art video action recognition models is encoded into video embeddings to relate seen and unseen activity classes. Experiments on three public datasets find that our approach outperforms other learnt semantic spaces, with an additional desirable feature of scalability, as recognition performance is seen to scale with the amount of data used. More generally, our results indicate that exploiting information from the video domain for IMU-based tasks is a promising direction, with tangible returns in a zero-shot learning scenario.

scholarly journals Mapping Distributional Semantics to Property Norms with Deep Neural Networks

2019 ◽  
Vol 3 (2) ◽  
pp. 30 ◽  
Author(s):  
Dandan Li ◽  
Douglas Summers-Stay
Keyword(s):  
Language Processing ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Feature Space ◽  
Semantic Space ◽  
Distributional Semantics ◽  
Word Embeddings ◽  
Different Types ◽  
Property Space ◽  
Psychology Literature

Word embeddings have been very successful in many natural language processing tasks, but they characterize the meaning of a word/concept by uninterpretable “context signatures”. Such a representation can render results obtained using embeddings difficult to interpret. Neighboring word vectors may have similar meanings, but in what way are they similar? That similarity may represent a synonymy, metonymy, or even antonymy relation. In the cognitive psychology literature, in contrast, concepts are frequently represented by their relations with properties. These properties are produced by test subjects when asked to describe important features of concepts. As such, they form a natural, intuitive feature space. In this work, we present a neural-network-based method for mapping a distributional semantic space onto a human-built property space automatically. We evaluate our method on word embeddings learned with different types of contexts, and report state-of-the-art performances on the widely used McRae semantic feature production norms.

scholarly journals Gaussian Embedding of Linked Documents from a Pretrained Semantic Space

2020 ◽  
Author(s):  
Antoine Gourru ◽  
Julien Velcin ◽  
Julien Jacques
Keyword(s):  
Vector Space ◽  
Gaussian Distribution ◽  
Link Prediction ◽  
State Of The Art ◽  
Semantic Space ◽  
Generative Model ◽  
New Method ◽  
Citation Networks ◽  
Word Embeddings ◽  
Art Methods

Gaussian Embedding of Linked Documents (GELD) is a new method that embeds linked documents (e.g., citation networks) onto a pretrained semantic space (e.g., a set of word embeddings). We formulate the problem in such a way that we model each document as a Gaussian distribution in the word vector space. We design a generative model that combines both words and links in a consistent way. Leveraging the variance of a document allows us to model the uncertainty related to word and link generation. In most cases, our method outperforms state-of-the-art methods when using our document vectors as features for usual downstream tasks. In particular, GELD achieves better accuracy in classification and link prediction on Cora and Dblp. In addition, we demonstrate qualitatively the convenience of several properties of our method. We provide the implementation of GELD and the evaluation datasets to the community (https://github.com/AntoineGourru/DNEmbedding).

Learning emotional word embeddings for sentiment analysis

2021 ◽  
pp. 1-13
Author(s):  
Qingtian Zeng ◽  
Xishi Zhao ◽  
Xiaohui Hu ◽  
Hua Duan ◽  
Zhongying Zhao ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Emotional Word ◽  
Classification Model ◽  
Data Sets ◽  
Word Embeddings ◽  
Real World Data ◽  
Text Documents

Word embeddings have been successfully applied in many natural language processing tasks due to its their effectiveness. However, the state-of-the-art algorithms for learning word representations from large amounts of text documents ignore emotional information, which is a significant research problem that must be addressed. To solve the above problem, we propose an emotional word embedding (EWE) model for sentiment analysis in this paper. This method first applies pre-trained word vectors to represent document features using two different linear weighting methods. Then, the resulting document vectors are input to a classification model and used to train a text sentiment classifier, which is based on a neural network. In this way, the emotional polarity of the text is propagated into the word vectors. The experimental results on three kinds of real-world data sets demonstrate that the proposed EWE model achieves superior performances on text sentiment prediction, text similarity calculation, and word emotional expression tasks compared to other state-of-the-art models.

Incorporating LDA With Word Embedding for Web Service Clustering

2018 ◽  
Vol 15 (4) ◽  
pp. 29-44 ◽  
Author(s):  
Yi Zhao ◽  
Chong Wang ◽  
Jian Wang ◽  
Keqing He
Keyword(s):  
Web Service ◽  
Service Discovery ◽  
Word Embedding ◽  
The Internet ◽  
Word Embeddings ◽  
Training Process ◽  
Web Service Discovery ◽  
Processing Data ◽  
Clustering Approach ◽  
Service Clustering

With the rapid growth of web services on the internet, web service discovery has become a hot topic in services computing. Faced with the heterogeneous and unstructured service descriptions, many service clustering approaches have been proposed to promote web service discovery, and many other approaches leveraged auxiliary features to enhance the classical LDA model to achieve better clustering performance. However, these extended LDA approaches still have limitations in processing data sparsity and noise words. This article proposes a novel web service clustering approach by incorporating LDA with word embedding, which leverages relevant words obtained based on word embedding to improve the performance of web service clustering. Especially, the semantically relevant words of service keywords by Word2vec were used to train the word embeddings and then incorporated into the LDA training process. Finally, experiments conducted on a real-world dataset published on ProgrammableWeb show that the authors' proposed approach can achieve better clustering performance than several classical approaches.

Sign in / Sign up

Export Citation Format

Share Document