scholarly journals RECAP reveals the true statistical significance of ChIP-seq peak calls

2019 ◽  
Vol 35 (19) ◽  
pp. 3592-3598 ◽  
Author(s):  
Justin G Chitpin ◽  
Aseel Awdeh ◽  
Theodore J Perkins
Keyword(s):  
False Discovery Rate ◽  
Statistical Significance ◽  
Statistical Hypothesis ◽  
Supplementary Information ◽  
Peak Calling ◽  
Statistical Hypothesis Testing ◽  
P Values ◽  
False Discovery ◽  
Genomic Regions ◽  
And Control

Abstract Motivation Chromatin Immunopreciptation (ChIP)-seq is used extensively to identify sites of transcription factor binding or regions of epigenetic modifications to the genome. A key step in ChIP-seq analysis is peak calling, where genomic regions enriched for ChIP versus control reads are identified. Many programs have been designed to solve this task, but nearly all fall into the statistical trap of using the data twice—once to determine candidate enriched regions, and again to assess enrichment by classical statistical hypothesis testing. This double use of the data invalidates the statistical significance assigned to enriched regions, thus the true significance or reliability of peak calls remains unknown. Results Using simulated and real ChIP-seq data, we show that three well-known peak callers, MACS, SICER and diffReps, output biased P-values and false discovery rate estimates that can be many orders of magnitude too optimistic. We propose a wrapper algorithm, RECAP, that uses resampling of ChIP-seq and control data to estimate a monotone transform correcting for biases built into peak calling algorithms. When applied to null hypothesis data, where there is no enrichment between ChIP-seq and control, P-values recalibrated by RECAP are approximately uniformly distributed. On data where there is genuine enrichment, RECAP P-values give a better estimate of the true statistical significance of candidate peaks and better false discovery rate estimates, which correlate better with empirical reproducibility. RECAP is a powerful new tool for assessing the true statistical significance of ChIP-seq peak calls. Availability and implementation The RECAP software is available through www.perkinslab.ca or on github at https://github.com/theodorejperkins/RECAP. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals RECAP reveals the true statistical significance of ChIP-seq peak calls

2018 ◽  
Author(s):  
Justin G. Chitpin ◽  
Aseel Awdeh ◽  
Theodore J. Perkins
Keyword(s):  
False Discovery Rate ◽  
Statistical Significance ◽  
Statistical Hypothesis ◽  
Data Sets ◽  
Peak Calling ◽  
Statistical Hypothesis Testing ◽  
P Values ◽  
False Discovery ◽  
Genomic Regions ◽  
And Control

AbstractMotivationChlP-seq is used extensively to identify sites of transcription factor binding or regions of epigenetic modifications to the genome. A key step in ChIP-seq analysis is peak calling, where genomic regions enriched for ChIP versus control reads are identified. Many programs have been designed to solve this task, but nearly all fall into the statistical trap of using the data twice—once to determine candidate enriched regions, and again to assess enrichment by classical statistical hypothesis testing. This double use of the data invalidates the statistical significance assigned to enriched regions, and as a consequence, invalidates false discovery rate estimates. Thus, the true significance or reliability of peak calls remains unknown.ResultsUsing simulated and real ChIP-seq data sets, we show that three well-known peak callers, MACS, SICER and diffReps, output optimistically biased p-values, and therefore optimistic false discovery rate estimates—in some cases, many orders of magnitude too optimistic. We propose a wrapper algorithm, RECAP, that uses resampling of ChIP-seq and control data to estimate and correct for biases built into peak calling algorithms. P-values recalibrated by RECAP are approximately uniformly distributed when applied to null hypothesis data, in which ChIP-seq and control come from the same genomic distributions. When applied to non-null data, RECAP p-values give a better estimate of the true statistical significance of candidate peaks and better false discovery rate estimates, which correlate better with empirical reproducibility. RECAP is a powerful new tool for assessing the true statistical significance of ChIP-seq peak calls.AvailabilityThe RECAP software is available on github at https://github.com/theodorejperkins/[email protected]

scholarly journals Making Decisions with Data: Understanding Hypothesis Testing & Statistical Significance

2019 ◽  
Vol 81 (8) ◽  
pp. 535-542
Author(s):  
Robert A. Cooper
Keyword(s):  
Hypothesis Testing ◽  
Statistical Methods ◽  
Scientific Literacy ◽  
Statistical Significance ◽  
Statistical Hypothesis ◽  
Statistical Hypothesis Testing ◽  
P Values ◽  
Using Data ◽  
Science Classes ◽  
Practice Of Science

Statistical methods are indispensable to the practice of science. But statistical hypothesis testing can seem daunting, with P-values, null hypotheses, and the concept of statistical significance. This article explains the concepts associated with statistical hypothesis testing using the story of “the lady tasting tea,” then walks the reader through an application of the independent-samples t-test using data from Peter and Rosemary Grant's investigations of Darwin's finches. Understanding how scientists use statistics is an important component of scientific literacy, and students should have opportunities to use statistical methods like this in their science classes.

scholarly journals sepal: identifying transcript profiles with spatial patterns by diffusion-based modeling

2021 ◽  
Author(s):  
Alma Andersson ◽  
Joakim Lundeberg
Keyword(s):  
Spatial Patterns ◽  
Expression Profiles ◽  
Synthetic Data ◽  
Real Data ◽  
Cell Types ◽  
Statistical Hypothesis ◽  
Supplementary Information ◽  
Statistical Hypothesis Testing ◽  
Transcriptomics Data ◽  
Transcript Profiles

Abstract Motivation Collection of spatial signals in large numbers has become a routine task in multiple omics-fields, but parsing of these rich datasets still pose certain challenges. In whole or near-full transcriptome spatial techniques, spurious expression profiles are intermixed with those exhibiting an organized structure. To distinguish profiles with spatial patterns from the background noise, a metric that enables quantification of spatial structure is desirable. Current methods designed for similar purposes tend to be built around a framework of statistical hypothesis testing, hence we were compelled to explore a fundamentally different strategy. Results We propose an unexplored approach to analyze spatial transcriptomics data, simulating diffusion of individual transcripts to extract genes with spatial patterns. The method performed as expected when presented with synthetic data. When applied to real data, it identified genes with distinct spatial profiles, involved in key biological processes or characteristic for certain cell types. Compared to existing methods, ours seemed to be less informed by the genes’ expression levels and showed better time performance when run with multiple cores. Availabilityand implementation Open-source Python package with a command line interface (CLI), freely available at https://github.com/almaan/sepal under an MIT licence. A mirror of the GitHub repository can be found at Zenodo, doi: 10.5281/zenodo.4573237. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals New mixture models for decoy-free false discovery rate estimation in mass spectrometry proteomics

2020 ◽  
Vol 36 (Supplement_2) ◽  
pp. i745-i753
Author(s):  
Yisu Peng ◽  
Shantanu Jain ◽  
Yong Fuga Li ◽  
Michal Greguš ◽  
Alexander R. Ivanov ◽  
...  
Keyword(s):  
Mass Spectrometry ◽  
False Discovery Rate ◽  
Mixture Models ◽  
Experimental Spectrum ◽  
Supplementary Information ◽  
Accurate Estimation ◽  
Component Mixture ◽  
Second Best ◽  
False Discovery ◽  
Score Distributions

Abstract Motivation Accurate estimation of false discovery rate (FDR) of spectral identification is a central problem in mass spectrometry-based proteomics. Over the past two decades, target-decoy approaches (TDAs) and decoy-free approaches (DFAs) have been widely used to estimate FDR. TDAs use a database of decoy species to faithfully model score distributions of incorrect peptide-spectrum matches (PSMs). DFAs, on the other hand, fit two-component mixture models to learn the parameters of correct and incorrect PSM score distributions. While conceptually straightforward, both approaches lead to problems in practice, particularly in experiments that push instrumentation to the limit and generate low fragmentation-efficiency and low signal-to-noise-ratio spectra. Results We introduce a new decoy-free framework for FDR estimation that generalizes present DFAs while exploiting more search data in a manner similar to TDAs. Our approach relies on multi-component mixtures, in which score distributions corresponding to the correct PSMs, best incorrect PSMs and second-best incorrect PSMs are modeled by the skew normal family. We derive EM algorithms to estimate parameters of these distributions from the scores of best and second-best PSMs associated with each experimental spectrum. We evaluate our models on multiple proteomics datasets and a HeLa cell digest case study consisting of more than a million spectra in total. We provide evidence of improved performance over existing DFAs and improved stability and speed over TDAs without any performance degradation. We propose that the new strategy has the potential to extend beyond peptide identification and reduce the need for TDA on all analytical platforms. Availabilityand implementation https://github.com/shawn-peng/FDR-estimation. Supplementary information Supplementary data are available at Bioinformatics online.

On “Field Significance” and the False Discovery Rate

2006 ◽  
Vol 45 (9) ◽  
pp. 1181-1189 ◽  
Author(s):  
D. S. Wilks
Keyword(s):  
False Discovery Rate ◽  
Statistical Power ◽  
Statistical Significance ◽  
Significance Test ◽  
P Value ◽  
Test Statistic ◽  
Global Test ◽  
Additional Advantage ◽  
Counting Procedure ◽  
False Discovery

Abstract The conventional approach to evaluating the joint statistical significance of multiple hypothesis tests (i.e., “field,” or “global,” significance) in meteorology and climatology is to count the number of individual (or “local”) tests yielding nominally significant results and then to judge the unusualness of this integer value in the context of the distribution of such counts that would occur if all local null hypotheses were true. The sensitivity (i.e., statistical power) of this approach is potentially compromised both by the discrete nature of the test statistic and by the fact that the approach ignores the confidence with which locally significant tests reject their null hypotheses. An alternative global test statistic that has neither of these problems is the minimum p value among all of the local tests. Evaluation of field significance using the minimum local p value as the global test statistic, which is also known as the Walker test, has strong connections to the joint evaluation of multiple tests in a way that controls the “false discovery rate” (FDR, or the expected fraction of local null hypothesis rejections that are incorrect). In particular, using the minimum local p value to evaluate field significance at a level αglobal is nearly equivalent to the slightly more powerful global test based on the FDR criterion. An additional advantage shared by Walker’s test and the FDR approach is that both are robust to spatial dependence within the field of tests. The FDR method not only provides a more broadly applicable and generally more powerful field significance test than the conventional counting procedure but also allows better identification of locations with significant differences, because fewer than αglobal × 100% (on average) of apparently significant local tests will have resulted from local null hypotheses that are true.

Multiple Hypothesis Testing for Data Mining

2011 ◽  
pp. 1390-1395
Author(s):  
Sach Mukherjee
Keyword(s):  
Data Mining ◽  
Hypothesis Testing ◽  
Conventional Treatment ◽  
Statistical Significance ◽  
Multiple Hypothesis Testing ◽  
Statistical Hypothesis ◽  
Single Variable ◽  
Statistical Hypothesis Testing ◽  
Computationally Efficient ◽  
Error Probabilities

A number of important problems in data mining can be usefully addressed within the framework of statistical hypothesis testing. However, while the conventional treatment of statistical significance deals with error probabilities at the level of a single variable, practical data mining tasks tend to involve thousands, if not millions, of variables. This Chapter looks at some of the issues that arise in the application of hypothesis tests to multi-variable data mining problems, and describes two computationally efficient procedures by which these issues can be addressed.

Multiple Hypothesis Testing for Data Mining

2011 ◽  
pp. 848-853
Author(s):  
Sach Mukherjee
Keyword(s):  
Data Mining ◽  
Hypothesis Testing ◽  
Conventional Treatment ◽  
Statistical Significance ◽  
Multiple Hypothesis Testing ◽  
Statistical Hypothesis ◽  
Single Variable ◽  
Statistical Hypothesis Testing ◽  
Computationally Efficient ◽  
Error Probabilities

A number of important problems in data mining can be usefully addressed within the framework of statistical hypothesis testing. However, while the conventional treatment of statistical significance deals with error probabilities at the level of a single variable, practical data mining tasks tend to involve thousands, if not millions, of variables. This Chapter looks at some of the issues that arise in the application of hypothesis tests to multi-variable data mining problems, and describes two computationally efficient procedures by which these issues can be addressed.

DiffNetFDR: differential network analysis with false discovery rate control

2019 ◽  
Vol 35 (17) ◽  
pp. 3184-3186
Author(s):  
Xiao-Fei Zhang ◽  
Le Ou-Yang ◽  
Shuo Yang ◽  
Xiaohua Hu ◽  
Hong Yan
Keyword(s):  
False Discovery Rate ◽  
Graphical Models ◽  
Biological Significance ◽  
R Package ◽  
Supplementary Information ◽  
Gaussian Graphical Models ◽  
Multiple Testing Procedure ◽  
False Discovery ◽  
Differential Network ◽  
Shiny App

Abstract Summary To identify biological network rewiring under different conditions, we develop a user-friendly R package, named DiffNetFDR, to implement two methods developed for testing the difference in different Gaussian graphical models. Compared to existing tools, our methods have the following features: (i) they are based on Gaussian graphical models which can capture the changes of conditional dependencies; (ii) they determine the tuning parameters in a data-driven manner; (iii) they take a multiple testing procedure to control the overall false discovery rate; and (iv) our approach defines the differential network based on partial correlation coefficients so that the spurious differential edges caused by the variants of conditional variances can be excluded. We also develop a Shiny application to provide easier analysis and visualization. Simulation studies are conducted to evaluate the performance of our methods. We also apply our methods to two real gene expression datasets. The effectiveness of our methods is validated by the biological significance of the identified differential networks. Availability and implementation R package and Shiny app are available at https://github.com/Zhangxf-ccnu/DiffNetFDR. Supplementary information Supplementary data are available at Bioinformatics online.

Sign in / Sign up

Export Citation Format

Share Document