Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model’s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications.

Objective bayesian analysis for multiple repairable systems

This article focus on the analysis of the reliability of multiple identical systems that can have multiple failures over time. A repairable system is defined as a system that can be restored to operating state in the event of a failure. This work under minimal repair, it is assumed that the failure has a power law intensity and the Bayesian approach is used to estimate the unknown parameters. The Bayesian estimators are obtained using two objective priors know as Jeffreys and reference priors. We proved that obtained reference prior is also a matching prior for both parameters, i.e., the credibility intervals have accurate frequentist coverage, while the Jeffreys prior returns unbiased estimates for the parameters. To illustrate the applicability of our Bayesian estimators, a new data set related to the failures of Brazilian sugar cane harvesters is considered.

Three essays on crime policy and the Bayesian bootstrap

This dissertation consists of three chapters. In the first chapter, I analyze credible intervals for quantiles constructed using Bayesian bootstrap techniques and show that credible intervals constructed using the "continuity-corrected" Bayesian bootstrap (Banks, 1988) have frequentist coverage probability error of only O(n [superscript -1]). In addition, I show that these "continuity-corrected" Bayesian bootstrap credible intervals achieve the same frequentist coverage probability as the frequentist confidence intervals of Goldman and Kaplan (2017), up to some error term of magnitude O(n [superscript -1]). Furthermore, I demonstrate that credible intervals constructed using the "continuity-corrected" Bayesian bootstrap have less frequentist coverage probability error than those constructed using the Bayesian bootstrap (Rubin, 1981). In the second chapter, I investigate three strikes laws, which mandate sharply increased sentences for criminals who commit a specific number of felonies. Specifically, I analyze the effect of these laws on violent crime rates using municipal-level data from the FBI. I compare violent crime rates of border municipalities in states with differing treatment statuses using a difference-in-differences specification with a sample matched on pre-treatment outcomes. I find no statistical evidence that three strikes laws reduce violent crime rates. I rule out reductions in violent crime rates greater than 1.3 [percent] and reject the hypothesis that three strikes laws reduce violent crime rates at the 5 [percent] significance level. Additional analyses and robustness checks support my main findings. In the third chapter, I examine medical marijuana laws (MMLs), which legalize the use, possession, and cultivation of marijuana by individuals with qualifying medical conditions. Namely, I employ municipal-level data from the FBI to analyze the effect of MMLs on violent crime rates. I compare municipalities in border regions with different treatments statuses using a difference-in-differences specification with a sample matched on pre-treatment outcomes. I find a lack of evidence for MMLs increasing violent crime rates, but I cannot eliminate the possibility of small-to-medium positive effects. However, I rule out increases in violent crime rates greater than 9.9 [percent] and reject the hypothesis that MMLs increase violent crime at the 10 [percent] significance level.

Calibrating general posterior credible regions

Summary Calibration of credible regions derived from under- or misspecified models is an important and challenging problem. In this paper, we introduce a scalar tuning parameter that controls the posterior distribution spread, and develop a Monte Carlo algorithm that sets this parameter so that the corresponding credible region achieves the nominal frequentist coverage probability.