The limits of distribution-free conditional predictive inference

We consider the problem of distribution-free predictive inference, with the goal of producing predictive coverage guarantees that hold conditionally rather than marginally. Existing methods such as conformal prediction offer marginal coverage guarantees, where predictive coverage holds on average over all possible test points, but this is not sufficient for many practical applications where we would like to know that our predictions are valid for a given individual, not merely on average over a population. On the other hand, exact conditional inference guarantees are known to be impossible without imposing assumptions on the underlying distribution. In this work, we aim to explore the space in between these two and examine what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.

The Precision of Gain Scores Under an Item Response Theory Perspective: A Comparison of Asymptotic and Exact Conditional Inference About Change

The precision of simple difference or “gain” scores is described in terms of their confidence intervals on the latent trait scale and of significance probabilities under the H₀ of no change. For this, two approaches are compared: one employs the asymptotic normal distribution of the maximum likelihood estimator of the person parameter, the other is based on the exact conditional distribution of the gain score, given the total number-correct score over the two time points. In either case, a detailed assessment of the precision of change measurements results. For illustration, results are presented of three test scales. The present methods yield more relevant and much more detailed psychometric information than the traditional estimation of reliability as a sole indicator of measurement precision. Other areas of application, namely, the comparison of the abilities of two examinees or the aggregation of individual signi.cance levels within groups of examinees, are also mentioned.