Predictive Uncertainty Estimation for Tractable Deep Probabilistic Models

Tractable Deep Probabilistic Models (TPMs) are generative models based on arithmetic circuits that allow for exact marginal inference in linear time. These models have obtained promising results in several machine learning tasks. Like many other models, TPMs can produce over-confident incorrect inferences, especially on regions with small statistical support. In this work, we will develop efficient estimators of the predictive uncertainty that are robust to data scarcity and outliers. We investigate two approaches. The first approach measures the variability of the output to perturbations of the model weights. The second approach captures the variability of the prediction to changes in the model architecture. We will evaluate the approaches on challenging tasks such as image completion and multilabel classification.

Bias due to Berkson error: issues when using predicted values in place of observed covariates

Summary Studies often want to test for the association between an unmeasured covariate and an outcome. In the absence of a measurement, the study may substitute values generated from a prediction model. Justification for such methods can be found by noting that, with standard assumptions, this is equivalent to fitting a regression model for an outcome variable when at least one covariate is measured with Berkson error. Under this setting, it is known that consistent or nearly consistent inference can be obtained under many linear and nonlinear outcome models. In this article, we focus on the linear regression outcome model and show that this consistency property does not hold when there is unmeasured confounding in the outcome model, in which case the marginal inference based on a covariate measured with Berkson error differs from the same inference based on observed covariates. Since unmeasured confounding is ubiquitous in applications, this severely limits the practical use of such measurements, and, in particular, the substitution of predicted values for observed covariates. These issues are illustrated using data from the National Health and Nutrition Examination Survey to study the joint association of total percent body fat and body mass index with HbA1c. It is shown that using predicted total percent body fat in place of observed percent body fat yields inferences which often differ significantly, in some cases suggesting opposite relationships among covariates.

Deep Convolutional Sum-Product Networks

We give conditions under which convolutional neural networks (CNNs) define valid sum-product networks (SPNs). One subclass, called convolutional SPNs (CSPNs), can be implemented using tensors, but also can suffer from being too shallow. Fortunately, tensors can be augmented while maintaining valid SPNs. This yields a larger subclass of CNNs, which we call deep convolutional SPNs (DCSPNs), where the convolutional and sum-pooling layers form rich directed acyclic graph structures. One salient feature of DCSPNs is that they are a rigorous probabilistic model. As such, they can exploit multiple kinds of probabilistic reasoning, including marginal inference and most probable explanation (MPE) inference. This allows an alternative method for learning DCSPNs using vectorized differentiable MPE, which plays a similar role to the generator in generative adversarial networks (GANs). Image sampling is yet another application demonstrating the robustness of DCSPNs. Our preliminary results on image sampling are encouraging, since the DCSPN sampled images exhibit variability. Experiments on image completion show that DCSPNs significantly outperform competing methods by achieving several state-of-the-art mean squared error (MSE) scores in both left-completion and bottom-completion in benchmark datasets.

Marginal Inference in Continuous Markov Random Fields Using Mixtures

Exact marginal inference in continuous graphical models is computationally challenging outside of a few special cases. Existing work on approximate inference has focused on approximately computing the messages as part of the loopy belief propagation algorithm either via sampling methods or moment matching relaxations. In this work, we present an alternative family of approximations that, instead of approximating the messages, approximates the beliefs in the continuous Bethe free energy using mixture distributions. We show that these types of approximations can be combined with numerical quadrature to yield algorithms with both theoretical guarantees on the quality of the approximation and significantly better practical performance in a variety of applications that are challenging for current state-of-the-art methods.

Estimating marginal and incremental effects in the analysis of medical expenditure panel data using marginalized two-part random-effects generalized Gamma models: Evidence from China healthcare cost data

Conditional two-part random-effects models have been proposed for the analysis of healthcare cost panel data that contain both zero costs from the non-users of healthcare facilities and positive costs from the users. These models have been extended to accommodate more flexible data structures when using the generalized Gamma distribution to model the positive healthcare expenditures. However, a major drawback with the extended model, which is inherited from the conditional models, is that it is fairly difficult to make direct marginal inference with respect to overall healthcare costs that includes both zeros and non-zeros, or even on positive healthcare costs. In this article, we first propose two types of marginalized two-part random-effects generalized Gamma models (m2RGGMs): Type I m2RGGMs for the inference on positive healthcare costs and Type II m2RGGMs for the inference on overall healthcare costs. Then, the concepts of marginal effect and incremental effect of a covariate on overall and positive healthcare costs are introduced, and estimation of these effects is carefully discussed. Especially, we derive the variance estimates of these effects by following the delta methods and Taylor series approximations for the purpose of making marginal inference. Parameter estimates of Type I and Type II m2RGGMs are obtained through maximum likelihood estimation. An empirical analysis of longitudinal healthcare costs collected in the China Health and Nutrition Survey is conducted using the proposed methodologies.

Direct and flexible marginal inference for semicontinuous data

The marginalized two-part (MTP) model for semicontinuous data proposed by Smith et al. provides direct inference for the effect of covariates on the marginal mean of positively continuous data with zeros. This brief note addresses mischaracterizations of the MTP model by Gebregziabher et al. Additionally, the MTP model is extended to incorporate the three-parameter generalized gamma distribution, which takes many well-known distributions as special cases, including the Weibull, gamma, inverse gamma, and log-normal distributions.