scholarly journals Meta-Amortized Variational Inference and Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 6404-6412
Author(s):  
Mike Wu ◽  
Kristy Choi ◽  
Noah Goodman ◽  
Stefano Ermon
Keyword(s):  
Image Classification ◽  
Probabilistic Models ◽  
Generative Models ◽  
Variational Inference ◽  
Inference Procedure ◽  
Data Transformations ◽  
Recent Success ◽  
Classification Tasks ◽  
Latent Representations ◽  
Inference Techniques

Despite the recent success in probabilistic modeling and their applications, generative models trained using traditional inference techniques struggle to adapt to new distributions, even when the target distribution may be closely related to the ones seen during training. In this work, we present a doubly-amortized variational inference procedure as a way to address this challenge. By sharing computation across not only a set of query inputs, but also a set of different, related probabilistic models, we learn transferable latent representations that generalize across several related distributions. In particular, given a set of distributions over images, we find the learned representations to transfer to different data transformations. We empirically demonstrate the effectiveness of our method by introducing the MetaVAE, and show that it significantly outperforms baselines on downstream image classification tasks on MNIST (10-50%) and NORB (10-35%).

scholarly journals Inference in probabilistic logic programs with continuous random variables

2012 ◽  
Vol 12 (4-5) ◽  
pp. 505-523 ◽  
Author(s):  
MUHAMMAD ASIFUL ISLAM ◽  
C. R. RAMAKRISHNAN ◽  
I. V. RAMAKRISHNAN
Keyword(s):  
Probabilistic Models ◽  
Random Variables ◽  
Evaluation Process ◽  
Probabilistic Logic ◽  
Logic Programs ◽  
Inference Procedure ◽  
Linear Equality Constraints ◽  
Query Answer ◽  
Inference Techniques ◽  
Special Case

AbstractProbabilistic Logic Programming (PLP), exemplified by Sato and Kameya's PRISM, Poole's ICL, Raedt et al.'s ProbLog and Vennekens et al.'s LPAD, is aimed at combining statistical and logical knowledge representation and inference. However, the inference techniques used in these works rely on enumerating sets of explanations for a query answer. Consequently, these languages permit very limited use of random variables with continuous distributions. In this paper, we present a symbolic inference procedure that uses constraints and represents sets of explanations without enumeration. This permits us to reason over PLPs with Gaussian or Gamma-distributed random variables (in addition to discrete-valued random variables) and linear equality constraints over reals. We develop the inference procedure in the context of PRISM; however the procedure's core ideas can be easily applied to other PLP languages as well. An interesting aspect of our inference procedure is that PRISM's query evaluation process becomes a special case in the absence of any continuous random variables in the program. The symbolic inference procedure enables us to reason over complex probabilistic models such as Kalman filters and a large subclass of Hybrid Bayesian networks that were hitherto not possible in PLP frameworks.

scholarly journals Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness

2021 ◽  
Author(s):  
Dazhong Shen ◽  
Chuan Qin ◽  
Chao Wang ◽  
Hengshu Zhu ◽  
Enhong Chen ◽  
...  
Keyword(s):  
Latent Variables ◽  
State Of The Art ◽  
Likelihood Estimation ◽  
Generative Models ◽  
Latent Space ◽  
Variational Autoencoder ◽  
Benchmark Datasets ◽  
Classification Tasks ◽  
Latent Representations ◽  
Low Uncertainty

As one of the most popular generative models, Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. However, when the decoder network is sufficiently expressive, VAE may lead to posterior collapse; that is, uninformative latent representations may be learned. To this end, in this paper, we propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space, and thus the representation can be learned in a meaningful and compact manner. Specifically, we first theoretically demonstrate that it will result in better latent space with high diversity and low uncertainty awareness by controlling the distribution of posterior’s parameters across the whole data accordingly. Then, without the introduction of new loss terms or modifying training strategies, we propose to exploit Dropout on the variances and Batch-Normalization on the means simultaneously to regularize their distributions implicitly. Furthermore, to evaluate the generalization effect, we also exploit DU-VAE for inverse autoregressive flow based-VAE (VAE-IAF) empirically. Finally, extensive experiments on three benchmark datasets clearly show that our approach can outperform state-of-the-art baselines on both likelihood estimation and underlying classification tasks.

scholarly journals Clustering-Based Relational Unsupervised Representation Learning with an Explicit Distributed Representation

2017 ◽  
Author(s):  
Sebastijan Dumancic ◽  
Hendrik Blockeel
Keyword(s):  
Relational Learning ◽  
Representation Learning ◽  
Relational Data ◽  
Distributed Representation ◽  
Learning Tasks ◽  
Wide Range ◽  
Lower Complexity ◽  
Classification Tasks ◽  
Latent Representations

The goal of unsupervised representation learning is to extract a new representation of data, such that solving many different tasks becomes easier. Existing methods typically focus on vectorized data and offer little support for relational data, which additionally describes relationships among instances. In this work we introduce an approach for relational unsupervised representation learning. Viewing a relational dataset as a hypergraph, new features are obtained by clustering vertices and hyperedges. To find a representation suited for many relational learning tasks, a wide range of similarities between relational objects is considered, e.g. feature and structural similarities. We experimentally evaluate the proposed approach and show that models learned on such latent representations perform better, have lower complexity, and outperform the existing approaches on classification tasks.

scholarly journals Predictive Uncertainty Estimation for Tractable Deep Probabilistic Models

2020 ◽  
Author(s):  
Julissa Villanueva Llerena
Keyword(s):  
Probabilistic Models ◽  
Linear Time ◽  
Generative Models ◽  
Uncertainty Estimation ◽  
Image Completion ◽  
Predictive Uncertainty ◽  
Learning Tasks ◽  
Challenging Tasks ◽  
Statistical Support ◽  
Marginal Inference

Tractable Deep Probabilistic Models (TPMs) are generative models based on arithmetic circuits that allow for exact marginal inference in linear time. These models have obtained promising results in several machine learning tasks. Like many other models, TPMs can produce over-confident incorrect inferences, especially on regions with small statistical support. In this work, we will develop efficient estimators of the predictive uncertainty that are robust to data scarcity and outliers. We investigate two approaches. The first approach measures the variability of the output to perturbations of the model weights. The second approach captures the variability of the prediction to changes in the model architecture. We will evaluate the approaches on challenging tasks such as image completion and multilabel classification.

scholarly journals Label Rectification Learning through Kernel Extreme Learning Machine

2021 ◽  
Vol 2021 ◽  
pp. 1-6
Author(s):  
Qiang Cai ◽  
Fenghai Li ◽  
Yifan Chen ◽  
Haisheng Li ◽  
Jian Cao ◽  
...  
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Image Classification ◽  
Extreme Learning Machine ◽  
Classification Performance ◽  
Considerable Progress ◽  
Strong Representation ◽  
Kernel Extreme Learning Machine ◽  
Classification Tasks ◽  
Learning Machine

Along with the strong representation of the convolutional neural network (CNN), image classification tasks have achieved considerable progress. However, majority of works focus on designing complicated and redundant architectures for extracting informative features to improve classification performance. In this study, we concentrate on rectifying the incomplete outputs of CNN. To be concrete, we propose an innovative image classification method based on Label Rectification Learning (LRL) through kernel extreme learning machine (KELM). It mainly consists of two steps: (1) preclassification, extracting incomplete labels through a pretrained CNN, and (2) label rectification, rectifying the generated incomplete labels by the KELM to obtain the rectified labels. Experiments conducted on publicly available datasets demonstrate the effectiveness of our method. Notably, our method is extensible which can be easily integrated with off-the-shelf networks for improving performance.

scholarly journals Applying LCS To Affective Image Classification In Spatial-Frequency Domain

2014 ◽  
Vol 4 (2) ◽  
pp. 99-123 ◽  
Author(s):  
Po-Ming Lee ◽  
Tzu-Chien Hsiao
Keyword(s):  
Pattern Recognition ◽  
Spatial Frequency ◽  
Image Classification ◽  
Frequency Domain ◽  
Model Building ◽  
Experimental Paradigm ◽  
Wide Range ◽  
Classification Tasks ◽  
Affective Image Classification ◽  
Spatial Frequency Domain

Abstract Recent studies have utilizes color, texture, and composition information of images to achieve affective image classification. However, the features related to spatial-frequency domain that were proven to be useful for traditional pattern recognition have not been tested in this field yet. Furthermore, the experiments conducted by previous studies are not internationally-comparable due to the experimental paradigm adopted. In addition, contributed by recent advances in methodology, that are, Hilbert-Huang Transform (HHT) (i.e. Empirical Mode Decomposition (EMD) and Hilbert Transform (HT)), the resolution of frequency analysis has been improved. Hence, the goal of this research is to achieve the affective image-classification task by adopting a standard experimental paradigm introduces by psychologists in order to produce international-comparable and reproducible results; and also to explore the affective hidden patterns of images in the spatial-frequency domain. To accomplish these goals, multiple human-subject experiments were conducted in laboratory. Extended Classifier Systems (XCSs) was used for model building because the XCS has been applied to a wide range of classification tasks and proved to be competitive in pattern recognition. To exploit the information in the spatial-frequency domain, the traditional EMD has been extended to a two-dimensional version. To summarize, the model built by using the XCS achieves Area Under Curve (AUC) = 0.91 and accuracy rate over 86%. The result of the XCS was compared with other traditional machine-learning algorithms (e.g., Radial-Basis Function Network (RBF Network)) that are normally used for classification tasks. Contributed by proper selection of features for model building, user-independent findings were obtained. For example, it is found that the horizontal visual stimulations contribute more to the emotion elicitation than the vertical visual stimulation. The effect of hue, saturation, and brightness; is also presented.

scholarly journals Evaluating probabilistic programming and fast variational Bayesian inference in phylogenetics

2019 ◽  
Author(s):  
Mathieu Fourment ◽  
Aaron E. Darling
Keyword(s):  
Probabilistic Models ◽  
Probability Distributions ◽  
Mean Field ◽  
Black Box ◽  
Variational Inference ◽  
Machine Learning Techniques ◽  
Mcmc Methods ◽  
Substitution Model ◽  
Probabilistic Programming ◽  
Phylogenetic Models

AbstractRecent advances in statistical machine learning techniques have led to the creation of probabilistic programming frameworks. These frameworks enable probabilistic models to be rapidly prototyped and fit to data using scalable approximation methods such as variational inference. In this work, we explore the use of the Stan language for probabilistic programming in application to phylogenetic models. We show that many commonly used phylogenetic models including the general time reversible (GTR) substitution model, rate heterogeneity among sites, and a range of coalescent models can be implemented using a probabilistic programming language. The posterior probability distributions obtained via the black box variational inference engine in Stan were compared to those obtained with reference implementations of Markov chain Monte Carlo (MCMC) for phylogenetic inference. We find that black box variational inference in Stan is less accurate than MCMC methods for phylogenetic models, but requires far less compute time. Finally, we evaluate a custom implementation of mean-field variational inference on the Jukes-Cantor substitution model and show that a specialized implementation of variational inference can be two orders of magnitude faster and more accurate than a general purpose probabilistic implementation.

scholarly journals RSI-CB: A Large-Scale Remote Sensing Image Classification Benchmark Using Crowdsourced Data

Sensors ◽  
2020 ◽  
Vol 20 (6) ◽  
pp. 1594
Author(s):  
Haifeng Li ◽  
Xin Dou ◽  
Chao Tao ◽  
Zhixiang Wu ◽  
Jie Chen ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Image Classification ◽  
Large Scale ◽  
Remote Sensing Image ◽  
National Standard ◽  
Natural Image ◽  
Deep Convolutional Neural Networks ◽  
Remote Sensing Image Classification ◽  
Crowdsourced Data ◽  
Classification Tasks

Image classification is a fundamental task in remote sensing image processing. In recent years, deep convolutional neural networks (DCNNs) have experienced significant breakthroughs in natural image recognition. The remote sensing field, however, is still lacking a large-scale benchmark similar to ImageNet. In this paper, we propose a remote sensing image classification benchmark (RSI-CB) based on massive, scalable, and diverse crowdsourced data. Using crowdsourced data, such as Open Street Map (OSM) data, ground objects in remote sensing images can be annotated effectively using points of interest, vector data from OSM, or other crowdsourced data. These annotated images can, then, be used in remote sensing image classification tasks. Based on this method, we construct a worldwide large-scale benchmark for remote sensing image classification. This benchmark has large-scale geographical distribution and large total image number. It contains six categories with 35 sub-classes of more than 24,000 images of size 256 × 256 pixels. This classification system of ground objects is defined according to the national standard of land-use classification in China and is inspired by the hierarchy mechanism of ImageNet. Finally, we conduct numerous experiments to compare RSI-CB with the SAT-4, SAT-6, and UC-Merced data sets. The experiments show that RSI-CB is more suitable as a benchmark for remote sensing image classification tasks than other benchmarks in the big data era and has many potential applications.

Sign in / Sign up

Export Citation Format

Share Document