Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing

Summary Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.

Permutation-based simultaneous confidence bounds for the false discovery proportion

SummaryWhen multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries is small with high confidence. In this paper, confidence upper bounds for the false discovery proportion are constructed, which are simultaneous over all rejection cut-offs. In particular, this allows the user to select a set of hypotheses post hoc such that the false discovery proportion lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen (2006) has developed an exact, permutation-based and computationally feasible method for obtaining simultaneous false discovery proportion bounds. We propose an exact method which uniformly improves that procedure. Further, we provide a generalization of the method that lets the user select the shape of the simultaneous confidence bounds; this gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as significance analysis of microarrays and the maxT method of Westfall & Young (1993), are obtained as special cases.