scholarly journals Clustering-Based Relational Unsupervised Representation Learning with an Explicit Distributed Representation

2017 ◽  
Author(s):  
Sebastijan Dumancic ◽  
Hendrik Blockeel
Keyword(s):  
Relational Learning ◽  
Representation Learning ◽  
Relational Data ◽  
Distributed Representation ◽  
Learning Tasks ◽  
Wide Range ◽  
Lower Complexity ◽  
Classification Tasks ◽  
Latent Representations

The goal of unsupervised representation learning is to extract a new representation of data, such that solving many different tasks becomes easier. Existing methods typically focus on vectorized data and offer little support for relational data, which additionally describes relationships among instances. In this work we introduce an approach for relational unsupervised representation learning. Viewing a relational dataset as a hypergraph, new features are obtained by clustering vertices and hyperedges. To find a representation suited for many relational learning tasks, a wide range of similarities between relational objects is considered, e.g. feature and structural similarities. We experimentally evaluate the proposed approach and show that models learned on such latent representations perform better, have lower complexity, and outperform the existing approaches on classification tasks.

scholarly journals Masking Orchestration: Multi-Task Pretraining for Multi-Role Dialogue Representation Learning

2020 ◽  
Vol 34 (05) ◽  
pp. 9217-9224
Author(s):  
Tianyi Wang ◽  
Yating Zhang ◽  
Xiaozhong Liu ◽  
Changlong Sun ◽  
Qiong Zhang
Keyword(s):  
Knowledge Integration ◽  
Question Answering ◽  
Representation Learning ◽  
Essential Information ◽  
External Knowledge ◽  
Learning Tasks ◽  
Wide Range ◽  
Specific Learning

Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types unsupervised pretraining tasks where the training objectives are given naturally according to the nature of the utterance and the structure of the multi-role conversation. Meanwhile, in order to locate essential information for dialogue summarization/extraction, the pretraining process enables external knowledge integration. The proposed fine-tuned pretraining mechanism is comprehensively evaluated via three different dialogue datasets along with a number of downstream dialogue-mining tasks. Result shows that the proposed pretraining mechanism significantly contributes to all the downstream tasks without discrimination to different encoders.

scholarly journals Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks

2018 ◽  
Author(s):  
Renjie Zheng ◽  
Junkun Chen ◽  
Xipeng Qiu
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Representation Learning ◽  
Training Data ◽  
Specific Information ◽  
Distributed Representation ◽  
Source Codes ◽  
Multiple Tasks ◽  
Classification Tasks

Distributed representation plays an important role in deep learning based natural language processing. However, the representation of a sentence often varies in different tasks, which is usually learned from scratch and suffers from the limited amounts of training data. In this paper, we claim that a good sentence representation should be invariant and can benefit the various subsequent tasks. To achieve this purpose, we propose a new scheme of information sharing for multi-task learning. More specifically, all tasks share the same sentence representation and each task can select the task-specific information from the shared sentence representation with attention mechanisms. The query vector of each task's attention could be either static parameters or generated dynamically. We conduct extensive experiments on 16 different text classification tasks, which demonstrate the benefits of our architecture. Source codes of this paper are available on Github.

scholarly journals A Self-Supervised Representation Learning of Sentence Structure for Authorship Attribution

2022 ◽  
Vol 16 (4) ◽  
pp. 1-16
Author(s):  
Fereshteh Jafariakinabad ◽  
Kien A. Hua
Keyword(s):  
Structural Information ◽  
Syntactic Structure ◽  
Representation Learning ◽  
Authorship Attribution ◽  
Sentence Structure ◽  
Vector Representation ◽  
Writing Style ◽  
Neural Models ◽  
Syntactic Information ◽  
Classification Tasks

The syntactic structure of sentences in a document substantially informs about its authorial writing style. Sentence representation learning has been widely explored in recent years and it has been shown that it improves the generalization of different downstream tasks across many domains. Even though utilizing probing methods in several studies suggests that these learned contextual representations implicitly encode some amount of syntax, explicit syntactic information further improves the performance of deep neural models in the domain of authorship attribution. These observations have motivated us to investigate the explicit representation learning of syntactic structure of sentences. In this article, we propose a self-supervised framework for learning structural representations of sentences. The self-supervised network contains two components; a lexical sub-network and a syntactic sub-network which take the sequence of words and their corresponding structural labels as the input, respectively. Due to the n -to-1 mapping of words to their structural labels, each word will be embedded into a vector representation which mainly carries structural information. We evaluate the learned structural representations of sentences using different probing tasks, and subsequently utilize them in the authorship attribution task. Our experimental results indicate that the structural embeddings significantly improve the classification tasks when concatenated with the existing pre-trained word embeddings.

scholarly journals PyDL8.5: a Library for Learning Optimal Decision Trees

2020 ◽  
Author(s):  
Gaël Aglin ◽  
Siegfried Nijssen ◽  
Pierre Schaus
Keyword(s):  
Machine Learning ◽  
Decision Trees ◽  
Efficient Algorithm ◽  
Optimal Decision ◽  
Learning Tasks ◽  
Explainable Ai ◽  
Classification Tasks ◽  
Interpretable Models ◽  
Limited Depth

Decision Trees (DTs) are widely used Machine Learning (ML) models with a broad range of applications. The interest in these models has increased even further in the context of Explainable AI (XAI), as decision trees of limited depth are very interpretable models. However, traditional algorithms for learning DTs are heuristic in nature; they may produce trees that are of suboptimal quality under depth constraints. We introduce PyDL8.5, a Python library to infer depth-constrained Optimal Decision Trees (ODTs). PyDL8.5 provides an interface for DL8.5, an efficient algorithm for inferring depth-constrained ODTs. The library provides an easy-to-use scikit-learn compatible interface. It cannot only be used for classification tasks, but also for regression, clustering, and other tasks. We introduce an interface that allows users to easily implement these other learning tasks. We provide a number of examples of how to use this library.

scholarly journals Applying LCS To Affective Image Classification In Spatial-Frequency Domain

2014 ◽  
Vol 4 (2) ◽  
pp. 99-123 ◽  
Author(s):  
Po-Ming Lee ◽  
Tzu-Chien Hsiao
Keyword(s):  
Pattern Recognition ◽  
Spatial Frequency ◽  
Image Classification ◽  
Frequency Domain ◽  
Model Building ◽  
Experimental Paradigm ◽  
Wide Range ◽  
Classification Tasks ◽  
Affective Image Classification ◽  
Spatial Frequency Domain

Abstract Recent studies have utilizes color, texture, and composition information of images to achieve affective image classification. However, the features related to spatial-frequency domain that were proven to be useful for traditional pattern recognition have not been tested in this field yet. Furthermore, the experiments conducted by previous studies are not internationally-comparable due to the experimental paradigm adopted. In addition, contributed by recent advances in methodology, that are, Hilbert-Huang Transform (HHT) (i.e. Empirical Mode Decomposition (EMD) and Hilbert Transform (HT)), the resolution of frequency analysis has been improved. Hence, the goal of this research is to achieve the affective image-classification task by adopting a standard experimental paradigm introduces by psychologists in order to produce international-comparable and reproducible results; and also to explore the affective hidden patterns of images in the spatial-frequency domain. To accomplish these goals, multiple human-subject experiments were conducted in laboratory. Extended Classifier Systems (XCSs) was used for model building because the XCS has been applied to a wide range of classification tasks and proved to be competitive in pattern recognition. To exploit the information in the spatial-frequency domain, the traditional EMD has been extended to a two-dimensional version. To summarize, the model built by using the XCS achieves Area Under Curve (AUC) = 0.91 and accuracy rate over 86%. The result of the XCS was compared with other traditional machine-learning algorithms (e.g., Radial-Basis Function Network (RBF Network)) that are normally used for classification tasks. Contributed by proper selection of features for model building, user-independent findings were obtained. For example, it is found that the horizontal visual stimulations contribute more to the emotion elicitation than the vertical visual stimulation. The effect of hue, saturation, and brightness; is also presented.

Examining Swarm Intelligence-based Feature Selection for Multi-Label Classification

2021 ◽  
Vol 2 (2) ◽  
Author(s):  
Awder Mohammed Ahmed ◽  
◽  
Adnan Mohsin Abdulazeez ◽  
Keyword(s):  
Feature Selection ◽  
Swarm Intelligence ◽  
Real World ◽  
Computational Cost ◽  
Computing Methods ◽  
Open Problems ◽  
Label Data ◽  
Filter Methods ◽  
Wide Range ◽  
Classification Tasks

Multi-label classification addresses the issues that more than one class label assigns to each instance. Many real-world multi-label classification tasks are high-dimensional due to digital technologies, leading to reduced performance of traditional multi-label classifiers. Feature selection is a common and successful approach to tackling this problem by retaining relevant features and eliminating redundant ones to reduce dimensionality. There is several feature selection that is successfully applied in multi-label learning. Most of those features are wrapper methods that employ a multi-label classifier in their processes. They run a classifier in each step, which requires a high computational cost, and thus they suffer from scalability issues. Filter methods are introduced to evaluate the feature subsets using information-theoretic mechanisms instead of running classifiers to deal with this issue. Most of the existing researches and review papers dealing with feature selection in single-label data. While, recently multi-label classification has a wide range of real-world applications such as image classification, emotion analysis, text mining, and bioinformatics. Moreover, researchers have recently focused on applying swarm intelligence methods in selecting prominent features of multi-label data. To the best of our knowledge, there is no review paper that reviews swarm intelligence-based methods for multi-label feature selection. Thus, in this paper, we provide a comprehensive review of different swarm intelligence and evolutionary computing methods of feature selection presented for multi-label classification tasks. To this end, in this review, we have investigated most of the well-known and state-of-the-art methods and categorize them based on different perspectives. We then provided the main characteristics of the existing multi-label feature selection techniques and compared them analytically. We also introduce benchmarks, evaluation measures, and standard datasets to facilitate research in this field. Moreover, we performed some experiments to compare existing works, and at the end of this survey, some challenges, issues, and open problems of this field are introduced to be considered by researchers in the future.

scholarly journals Graph representation learning: a survey

2020 ◽  
Vol 9 ◽  
Author(s):  
Fenxiao Chen ◽  
Yun-Cheng Wang ◽  
Bin Wang ◽  
C.-C. Jay Kuo
Keyword(s):  
Graph Embedding ◽  
Large Data ◽  
Representation Learning ◽  
Graph Representation ◽  
Data Sets ◽  
Graph Data ◽  
Graph Properties ◽  
Wide Range ◽  
Regular Lattices ◽  
Low Dimensional

Abstract Research on graph representation learning has received great attention in recent years since most data in real-world applications come in the form of graphs. High-dimensional graph data are often in irregular forms. They are more difficult to analyze than image/video/audio data defined on regular lattices. Various graph embedding techniques have been developed to convert the raw graph data into a low-dimensional vector representation while preserving the intrinsic graph properties. In this review, we first explain the graph embedding task and its challenges. Next, we review a wide range of graph embedding techniques with insights. Then, we evaluate several stat-of-the-art methods against small and large data sets and compare their performance. Finally, potential applications and future directions are presented.

scholarly journals Exponential Family Graph Embeddings

2020 ◽  
Vol 34 (04) ◽  
pp. 3357-3364
Author(s):  
Abdulkadir Celikkanat ◽  
Fragkiskos D. Malliaros
Keyword(s):  
Random Walk ◽  
Exponential Family ◽  
Representation Learning ◽  
Learning Problems ◽  
Interaction Patterns ◽  
Network Representation ◽  
Learning Tasks ◽  
Learning Techniques ◽  
Real World Datasets ◽  
Low Dimensional

Representing networks in a low dimensional latent space is a crucial task with many interesting applications in graph learning problems, such as link prediction and node classification. A widely applied network representation learning paradigm is based on the combination of random walks for sampling context nodes and the traditional Skip-Gram model to capture center-context node relationships. In this paper, we emphasize on exponential family distributions to capture rich interaction patterns between nodes in random walk sequences. We introduce the generic exponential family graph embedding model, that generalizes random walk-based network representation learning techniques to exponential family conditional distributions. We study three particular instances of this model, analyzing their properties and showing their relationship to existing unsupervised learning models. Our experimental evaluation on real-world datasets demonstrates that the proposed techniques outperform well-known baseline methods in two downstream machine learning tasks.

scholarly journals Affective Latent Representation of Acoustic and Lexical Features for Emotion Recognition

Sensors ◽  
2020 ◽  
Vol 20 (9) ◽  
pp. 2614
Author(s):  
Eesung Kim ◽  
Hyungchan Song ◽  
Jong Won Shin
Keyword(s):  
Emotion Recognition ◽  
Motion Capture ◽  
Deep Neural Network ◽  
Previous Attempt ◽  
Distributed Representation ◽  
Acoustic Features ◽  
Recognition Method ◽  
Latent Representations ◽  
Discriminant Power ◽  
Statistical Functionals

In this paper, we propose a novel emotion recognition method based on the underlying emotional characteristics extracted from a conditional adversarial auto-encoder (CAAE), in which both acoustic and lexical features are used as inputs. The acoustic features are generated by calculating statistical functionals of low-level descriptors and by a deep neural network (DNN). These acoustic features are concatenated with three types of lexical features extracted from the text, which are a sparse representation, a distributed representation, and an affective lexicon-based dimensions. Two-dimensional latent representations similar to vectors in the valence-arousal space are obtained by a CAAE, which can be directly mapped into the emotional classes without the need for a sophisticated classifier. In contrast to the previous attempt to a CAAE using only acoustic features, the proposed approach could enhance the performance of the emotion recognition because combined acoustic and lexical features provide enough discriminant power. Experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpus showed that our method outperformed the previously reported best results on the same corpus, achieving 76.72% in the unweighted average recall.

scholarly journals Tile2Vec: Unsupervised Representation Learning for Spatially Distributed Data

2019 ◽  
Vol 33 ◽  
pp. 3967-3974 ◽  
Author(s):  
Neal Jean ◽  
Sherrie Wang ◽  
Anshul Samar ◽  
George Azzari ◽  
David Lobell ◽  
...  
Keyword(s):  
Natural Language ◽  
Learning Algorithm ◽  
Representation Learning ◽  
Distributed Data ◽  
Simple Arithmetic ◽  
Latent Space ◽  
Wide Range ◽  
Spatially Distributed ◽  
Vector Representations ◽  
Spatially Distributed Data

Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distributional hypothesis from natural language — words appearing in similar contexts tend to have similar meanings — to spatially distributed data. We demonstrate empirically that Tile2Vec learns semantically meaningful representations for both image and non-image datasets. Our learned representations significantly improve performance in downstream classification tasks and, similarly to word vectors, allow visual analogies to be obtained via simple arithmetic in the latent space.

Sign in / Sign up

Export Citation Format

Share Document