Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing

Summary Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.

Permutation-based simultaneous confidence bounds for the false discovery proportion

SummaryWhen multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries is small with high confidence. In this paper, confidence upper bounds for the false discovery proportion are constructed, which are simultaneous over all rejection cut-offs. In particular, this allows the user to select a set of hypotheses post hoc such that the false discovery proportion lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen (2006) has developed an exact, permutation-based and computationally feasible method for obtaining simultaneous false discovery proportion bounds. We propose an exact method which uniformly improves that procedure. Further, we provide a generalization of the method that lets the user select the shape of the simultaneous confidence bounds; this gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as significance analysis of microarrays and the maxT method of Westfall & Young (1993), are obtained as special cases.

A pseudo knockoff filter for correlated features

In Barber & Candès (2015, Ann. Statist., 43, 2055–2085), the authors introduced a new variable selection procedure called the knockoff filter to control the false discovery rate (FDR) and proved that this method achieves exact FDR control. Inspired by the work by Barber & Candès (2015, Ann. Statist., 43, 2055–2085), we propose a pseudo knockoff filter that inherits some advantages of the original knockoff filter and has more flexibility in constructing its knockoff matrix. Moreover, we perform a number of numerical experiments that seem to suggest that the pseudo knockoff filter with the half Lasso statistic has FDR control and offers more power than the original knockoff filter with the Lasso Path or the half Lasso statistic for the numerical examples that we consider in this paper. Although we cannot establish rigourous FDR control for the pseudo knockoff filter, we provide some partial analysis of the pseudo knockoff filter with the half Lasso statistic and establish a uniform false discovery proportion bound and an expectation inequality.