AbstractMotivationChlP-seq is used extensively to identify sites of transcription factor binding or regions of epigenetic modifications to the genome. A key step in ChIP-seq analysis is peak calling, where genomic regions enriched for ChIP versus control reads are identified. Many programs have been designed to solve this task, but nearly all fall into the statistical trap of using the data twice—once to determine candidate enriched regions, and again to assess enrichment by classical statistical hypothesis testing. This double use of the data invalidates the statistical significance assigned to enriched regions, and as a consequence, invalidates false discovery rate estimates. Thus, the true significance or reliability of peak calls remains unknown.ResultsUsing simulated and real ChIP-seq data sets, we show that three well-known peak callers, MACS, SICER and diffReps, output optimistically biased p-values, and therefore optimistic false discovery rate estimates—in some cases, many orders of magnitude too optimistic. We propose a wrapper algorithm, RECAP, that uses resampling of ChIP-seq and control data to estimate and correct for biases built into peak calling algorithms. P-values recalibrated by RECAP are approximately uniformly distributed when applied to null hypothesis data, in which ChIP-seq and control come from the same genomic distributions. When applied to non-null data, RECAP p-values give a better estimate of the true statistical significance of candidate peaks and better false discovery rate estimates, which correlate better with empirical reproducibility. RECAP is a powerful new tool for assessing the true statistical significance of ChIP-seq peak calls.AvailabilityThe RECAP software is available on github at https://github.com/theodorejperkins/[email protected]
Making Decisions with Data: Understanding Hypothesis Testing & Statistical Significance
