scholarly journals A structured distributional model of sentence meaning and processing

2019 ◽  
Vol 25 (4) ◽  
pp. 483-502 ◽  
Author(s):  
E. Chersoni ◽  
E. Santus ◽  
L. Pannitto ◽  
A. Lenci ◽  
P. Blache ◽  
...  
Keyword(s):  
Semantic Representation ◽  
Formal Semantics ◽  
Formal Structure ◽  
Data Sets ◽  
Word Embeddings ◽  
Discourse Representation Theory ◽  
Event Knowledge ◽  
Sentence Meaning ◽  
Generalized Knowledge ◽  
Distributional Semantic Models

AbstractMost compositional distributional semantic models represent sentence meaning with a single vector. In this paper, we propose a structured distributional model (SDM) that combines word embeddings with formal semantics and is based on the assumption that sentences represent events and situations. The semantic representation of a sentence is a formal structure derived from discourse representation theory and containing distributional vectors. This structure is dynamically and incrementally built by integrating knowledge about events and their typical participants, as they are activated by lexical items. Event knowledge is modelled as a graph extracted from parsed corpora and encoding roles and relationships between participants that are represented as distributional vectors. SDM is grounded on extensive psycholinguistic research showing that generalized knowledge about events stored in semantic memory plays a key role in sentence comprehension.We evaluate SDMon two recently introduced compositionality data sets, and our results show that combining a simple compositionalmodel with event knowledge constantly improves performances, even with dif ferent types of word embeddings.

scholarly journals Investigating the impact of pre-processing techniques and pre-trained word embeddings in detecting Arabic health information on social media

2021 ◽  
Vol 8 (1) ◽  
Author(s):  
Yahya Albalawi ◽  
Jim Buckley ◽  
Nikola S. Nikolov
Keyword(s):  
Social Media ◽  
Deep Learning ◽  
Comprehensive Evaluation ◽  
Classification Problem ◽  
Data Sets ◽  
Word Embeddings ◽  
Data Set ◽  
Lower Accuracy ◽  
Health Related ◽  
The Impact

AbstractThis paper presents a comprehensive evaluation of data pre-processing and word embedding techniques in the context of Arabic document classification in the domain of health-related communication on social media. We evaluate 26 text pre-processings applied to Arabic tweets within the process of training a classifier to identify health-related tweets. For this task we use the (traditional) machine learning classifiers KNN, SVM, Multinomial NB and Logistic Regression. Furthermore, we report experimental results with the deep learning architectures BLSTM and CNN for the same text classification problem. Since word embeddings are more typically used as the input layer in deep networks, in the deep learning experiments we evaluate several state-of-the-art pre-trained word embeddings with the same text pre-processing applied. To achieve these goals, we use two data sets: one for both training and testing, and another for testing the generality of our models only. Our results point to the conclusion that only four out of the 26 pre-processings improve the classification accuracy significantly. For the first data set of Arabic tweets, we found that Mazajak CBOW pre-trained word embeddings as the input to a BLSTM deep network led to the most accurate classifier with F1 score of 89.7%. For the second data set, Mazajak Skip-Gram pre-trained word embeddings as the input to BLSTM led to the most accurate model with F1 score of 75.2% and accuracy of 90.7% compared to F1 score of 90.8% achieved by Mazajak CBOW for the same architecture but with lower accuracy of 70.89%. Our results also show that the performance of the best of the traditional classifier we trained is comparable to the deep learning methods on the first dataset, but significantly worse on the second dataset.

Learning emotional word embeddings for sentiment analysis

2021 ◽  
pp. 1-13
Author(s):  
Qingtian Zeng ◽  
Xishi Zhao ◽  
Xiaohui Hu ◽  
Hua Duan ◽  
Zhongying Zhao ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Emotional Word ◽  
Classification Model ◽  
Data Sets ◽  
Word Embeddings ◽  
Real World Data ◽  
Text Documents

Word embeddings have been successfully applied in many natural language processing tasks due to its their effectiveness. However, the state-of-the-art algorithms for learning word representations from large amounts of text documents ignore emotional information, which is a significant research problem that must be addressed. To solve the above problem, we propose an emotional word embedding (EWE) model for sentiment analysis in this paper. This method first applies pre-trained word vectors to represent document features using two different linear weighting methods. Then, the resulting document vectors are input to a classification model and used to train a text sentiment classifier, which is based on a neural network. In this way, the emotional polarity of the text is propagated into the word vectors. The experimental results on three kinds of real-world data sets demonstrate that the proposed EWE model achieves superior performances on text sentiment prediction, text similarity calculation, and word emotional expression tasks compared to other state-of-the-art models.

scholarly journals Unsupervised Compositionality Prediction of Nominal Compounds

2019 ◽  
Vol 45 (1) ◽  
pp. 1-57 ◽  
Author(s):  
Silvio Cordeiro ◽  
Aline Villavicencio ◽  
Marco Idiart ◽  
Carlos Ramisch
Keyword(s):  
Morphological Variation ◽  
Red Wine ◽  
Data Sets ◽  
High Agreement ◽  
Semantic Models ◽  
Corpus Size ◽  
Nominal Compounds ◽  
Distributional Semantic Models ◽  
The Impact

Nominal compounds such as red wine and nut case display a continuum of compositionality, with varying contributions from the components of the compound to its semantics. This article proposes a framework for compound compositionality prediction using distributional semantic models, evaluating to what extent they capture idiomaticity compared to human judgments. For evaluation, we introduce data sets containing human judgments in three languages: English, French, and Portuguese. The results obtained reveal a high agreement between the models and human predictions, suggesting that they are able to incorporate information about idiomaticity. We also present an in-depth evaluation of various factors that can affect prediction, such as model and corpus parameters and compositionality operations. General crosslingual analyses reveal the impact of morphological variation and corpus size in the ability of the model to predict compositionality, and of a uniform combination of the components for best results.

scholarly journals Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity

2020 ◽  
pp. 1-51
Author(s):  
Ivan Vulić ◽  
Simon Baker ◽  
Edoardo Maria Ponti ◽  
Ulla Petti ◽  
Ira Leviant ◽  
...  
Keyword(s):  
Semantic Similarity ◽  
Large Scale ◽  
Representation Learning ◽  
Data Sets ◽  
Word Embeddings ◽  
Data Set ◽  
Lexical Representations ◽  
Language Data ◽  
Weakly Supervised ◽  
Cross Lingual

We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering data sets for 12 typologically diverse languages, including major languages (e.g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e.g., Welsh, Kiswahili). Each language data set is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs, providing a representative coverage of word classes (nouns, verbs, adjectives, adverbs), frequency ranks, similarity intervals, lexical fields, and concreteness levels. Additionally, owing to the alignment of concepts across languages, we provide a suite of 66 crosslingual semantic similarity data sets. Because of its extensive size and language coverage, Multi-SimLex provides entirely novel opportunities for experimental evaluation and analysis. On its monolingual and crosslingual benchmarks, we evaluate and analyze a wide array of recent state-of-the-art monolingual and crosslingual representation models, including static and contextualized word embeddings (such as fastText, monolingual and multilingual BERT, XLM), externally informed lexical representations, as well as fully unsupervised and (weakly) supervised crosslingual word embeddings. We also present a step-by-step data set creation protocol for creating consistent, Multi-Simlex -style resources for additional languages.We make these contributions—the public release of Multi-SimLex data sets, their creation protocol, strong baseline results, and in-depth analyses which can be be helpful in guiding future developments in multilingual lexical semantics and representation learning—available via aWeb site that will encourage community effort in further expansion of Multi-Simlex to many more languages. Such a large-scale semantic resource could inspire significant further advances in NLP across languages.

scholarly journals Can prediction-based distributional semantic models predict typicality?

2019 ◽  
Vol 72 (8) ◽  
pp. 2084-2109
Author(s):  
Tom Heyman ◽  
Geert Heyman
Keyword(s):  
Computational Linguistics ◽  
Predictive Models ◽  
Primary Concern ◽  
Word Embeddings ◽  
Future Directions ◽  
Continuous Space ◽  
Text Collections ◽  
Similarity Ratings ◽  
Priming Word ◽  
Distributional Semantic Models

Recent advances in the field of computational linguistics have led to the development of various prediction-based models of semantics. These models seek to infer word representations from large text collections by predicting target words from neighbouring words (or vice versa). The resulting representations are vectors in a continuous space, collectively called word embeddings. Although psychological plausibility was not a primary concern for the developers of predictive models, it has been the topic of several recent studies in the field of psycholinguistics. That is, word embeddings have been linked to similarity ratings, word associations, semantic priming, word recognition latencies, and so on. Here, we build on this work by investigating category structure. Throughout seven experiments, we sought to predict human typicality judgements from two languages, Dutch and English, using different semantic spaces. More specifically, we extracted a number of predictor variables, and evaluated how well they could capture the typicality gradient of common categories (e.g., birds, fruit, vehicles, etc.). Overall, the performance of predictive models was rather modest and did not compare favourably with that of an older count-based model. These results are somewhat disappointing given the enthusiasm surrounding predictive models. Possible explanations and future directions are discussed.

scholarly journals Can prediction-based distributional semantic models predict typicality?

2018 ◽  
Author(s):  
Tom Heyman ◽  
Geert Heyman
Keyword(s):  
Computational Linguistics ◽  
Predictive Models ◽  
Primary Concern ◽  
Word Embeddings ◽  
Future Directions ◽  
Continuous Space ◽  
Text Collections ◽  
Similarity Ratings ◽  
Priming Word ◽  
Distributional Semantic Models

Recent advances in the field of computational linguistics have led to the development of various prediction-based models of semantics. These models seek to infer wordrepresentations from large text collections by predicting target words from neighboring words (or vice versa). The resulting representations are vectors in a continuous space, collectively called word embeddings. Although psychological plausibility was not a primary concern for the developers of predictive models, it has been the topic of several recent studies in the field of psycholinguistics. That is, word embeddings have been linked to similarity ratings, word associations, semantic priming, word recognition latencies, etcetera. Here, we build on this work by investigating category structure. Throughout seven experiments, we sought to predict human typicality judgments from two languages, Dutch and English, using different semantic spaces. More specifically, we extracted a number of predictor variables, and evaluated how well they could capture the typicality gradient of common categories (e.g., birds, fruit, vehicles,. . . ). Overall, the performance of predictive models was rather modest, and did not compare favorably to that of an older count-based model. These results are somewhat disappointing given the enthusiasm surrounding predictive models. Possible explanations and future directions are discussed.

scholarly journals Using the Outlier Detection Task to Evaluate Distributional Semantic Models

2018 ◽  
Vol 1 (1) ◽  
pp. 211-223
Author(s):  
Pablo Gamallo
Keyword(s):  
Outlier Detection ◽  
Detection Task ◽  
Bag Of Words ◽  
Word Embeddings ◽  
Specific Task ◽  
Semantic Models ◽  
Distributional Semantic Models ◽  
Syntactic Dependencies ◽  
Better Than

In this article, we define the outlier detection task and use it to compare neural-based word embeddings with transparent count-based distributional representations. Using the English Wikipedia as a text source to train the models, we observed that embeddings outperform count-based representations when their contexts are made up of bag-of-words. However, there are no sharp differences between the two models if the word contexts are defined as syntactic dependencies. In general, syntax-based models tend to perform better than those based on bag-of-words for this specific task. Similar experiments were carried out for Portuguese with similar results. The test datasets we have created for the outlier detection task in English and Portuguese are freely available.

scholarly journals Unsupervised Word Translation with Adversarial Autoencoder

2020 ◽  
Vol 46 (2) ◽  
pp. 257-288
Author(s):  
Tasnim Mohiuddin ◽  
Shafiq Joty
Keyword(s):  
Machine Translation ◽  
Superior Performance ◽  
Data Sets ◽  
Word Embeddings ◽  
Shared Space ◽  
Parallel Data ◽  
Adversarial Training ◽  
Word Translation ◽  
Input Reconstruction ◽  
Adversarial Model

Crosslingual word embeddings learned from monolingual embeddings have a crucial role in many downstream tasks, ranging from machine translation to transfer learning. Adversarial training has shown impressive success in learning crosslingual embeddings and the associated word translation task without any parallel data by mapping monolingual embeddings to a shared space. However, recent work has shown superior performance for non-adversarial methods in more challenging language pairs. In this article, we investigate adversarial autoencoder for unsupervised word translation and propose two novel extensions to it that yield more stable training and improved results. Our method includes regularization terms to enforce cycle consistency and input reconstruction, and puts the target encoders as an adversary against the corresponding discriminator. We use two types of refinement procedures sequentially after obtaining the trained encoders and mappings from the adversarial training, namely, refinement with Procrustes solution and refinement with symmetric re-weighting. Extensive experimentations with high- and low-resource languages from two different data sets show that our method achieves better performance than existing adversarial and non-adversarial approaches and is also competitive with the supervised system. Along with performing comprehensive ablation studies to understand the contribution of different components of our adversarial model, we also conduct a thorough analysis of the refinement procedures to understand their effects.

Sign in / Sign up

Export Citation Format

Share Document