Probabilistic forecasting of replication studies

Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.

Download Full-text

A study on the statistical significance of mutual information between morphology of a galaxy and its large-scale environment

Monthly Notices of the Royal Astronomical Society ◽

10.1093/mnras/staa2236 ◽

2020 ◽

Vol 497 (4) ◽

pp. 4077-4090 ◽

Cited By ~ 1

Author(s):

Suman Sarkar ◽

Biswajit Pandey

Keyword(s):

Mutual Information ◽

Large Scale ◽

Statistical Significance ◽

Sloan Digital Sky Survey ◽

Data Sets ◽

Information Theoretic ◽

Galaxy Distribution ◽

Sky Survey ◽

The Galaxy ◽

Physical Correlations

ABSTRACT A non-zero mutual information between morphology of a galaxy and its large-scale environment is known to exist in Sloan Digital Sky Survey (SDSS) upto a few tens of Mpc. It is important to test the statistical significance of these mutual information if any. We propose three different methods to test the statistical significance of these non-zero mutual information and apply them to SDSS and Millennium run simulation. We randomize the morphological information of SDSS galaxies without affecting their spatial distribution and compare the mutual information in the original and randomized data sets. We also divide the galaxy distribution into smaller subcubes and randomly shuffle them many times keeping the morphological information of galaxies intact. We compare the mutual information in the original SDSS data and its shuffled realizations for different shuffling lengths. Using a t-test, we find that a small but statistically significant (at $99.9{{\ \rm per\ cent}}$ confidence level) mutual information between morphology and environment exists upto the entire length-scale probed. We also conduct another experiment using mock data sets from a semi-analytic galaxy catalogue where we assign morphology to galaxies in a controlled manner based on the density at their locations. The experiment clearly demonstrates that mutual information can effectively capture the physical correlations between morphology and environment. Our analysis suggests that physical association between morphology and environment may extend to much larger length-scales than currently believed, and the information theoretic framework presented here can serve as a sensitive and useful probe of the assembly bias and large-scale environmental dependence of galaxy properties.

Download Full-text

eNetXplorer: an R package for the quantitative exploration of elastic net families for generalized linear models

10.1101/305870 ◽

2018 ◽

Cited By ~ 1

Author(s):

Julián Candia ◽

John S. Tsang

Keyword(s):

Generalized Linear Models ◽

Linear Models ◽

Statistical Significance ◽

Model Fitting ◽

Predictive Performance ◽

R Package ◽

Research Area ◽

Elastic Net ◽

Feature Identification ◽

Practical Applications

AbstractBackgroundRegularized generalized linear models (GLMs) are popular regression methods in bioinformatics, particularly useful in scenarios with fewer observations than parameters/features or when many of the features are correlated. In both ridge and lasso regularization, feature shrinkage is controlled by a penalty parameter λ. The elastic net introduces a mixing parameter α to tune the shrinkage continuously from ridge to lasso. Selecting α objectively and determining which features contributed significantly to prediction after model fitting remain a practical challenge given the paucity of available software to evaluate performance and statistical significance.ResultseNetXplorer builds on top of glmnet to address the above issues for linear (Gaussian), binomial (logistic), and multinomial GLMs. It provides new functionalities to empower practical applications by using a cross validation framework that assesses the predictive performance and statistical significance of a family of elastic net models (as α is varied) and of the corresponding features that contribute to prediction. The user can select which quality metrics to use to quantify the concordance between predicted and observed values, with defaults provided for each GLM. Statistical significance for each model (as defined by α) is determined based on comparison to a set of null models generated by random permutations of the response; the same permutation-based approach is used to evaluate the significance of individual features. In the analysis of large and complex biological datasets, such as transcriptomic and proteomic data, eNetXplorer provides summary statistics, output tables, and visualizations to help assess which subset(s) of features have predictive value for a set of response measurements, and to what extent those subset(s) of features can be expanded or reduced via regularization.ConclusionsThis package presents a framework and software for exploratory data analysis and visualization. By making regularized GLMs more accessible and interpretable, eNetXplorer guides the process to generate hypotheses based on features significantly associated with biological phenotypes of interest, e.g. to identify biomarkers for therapeutic responsiveness. eNetXplorer is also generally applicable to any research area that may benefit from predictive modeling and feature identification using regularized GLMs.Availability and implementationThe package is available under GPL-3 license at the CRAN repository, https://CRAN.R-project.org/package=eNetXplorer

Download Full-text

Mercator: An R Package for Visualization of Distance Matrices

10.1101/733261 ◽

2019 ◽

Author(s):

Zachary B. Abrams ◽

Caitlin E. Coombes ◽

Suli Li ◽

Kevin R. Coombes

Keyword(s):

Large Scale ◽

R Package ◽

Supplementary Information ◽

Data Sets ◽

Data Types ◽

Vast Number ◽

Large Scale Data ◽

Scientific Disciplines ◽

Compare And Contrast ◽

Scale Data

AbstractSummaryUnsupervised data analysis in many scientific disciplines is based on calculating distances between observations and finding ways to visualize those distances. These kinds of unsupervised analyses help researchers uncover patterns in large-scale data sets. However, researchers can select from a vast number of different distance metrics, each designed to highlight different aspects of different data types. There are also numerous visualization methods with their own strengths and weaknesses. To help researchers perform unsupervised analyses, we developed the Mercator R package. Mercator enables users to see important patterns in their data by generating multiple visualizations using different standard algorithms, making it particularly easy to compare and contrast the results arising from different metrics. By allowing users to select the distance metric that best fits their needs, Mercator helps researchers perform unsupervised analyses that use pattern identification through computation and visual inspection.Availability and ImplementationMercator is freely available at the Comprehensive R Archive Network (https://cran.r-project.org/web/packages/Mercator/index.html)[email protected] informationSupplementary data are available at Bioinformatics online.

Download Full-text

Probabilistic Forecasting and Model Validation for the First-Eocene Large-Scale Pilot Steamflood, Partitioned Zone, Saudi Arabia and Kuwait

SPE Reservoir Evaluation & Engineering ◽

10.2118/150580-pa ◽

2013 ◽

Vol 16 (01) ◽

pp. 97-116 ◽

Cited By ~ 2

Author(s):

W. Terry Osterloh ◽

Don S. Mims ◽

W. Scott Meddaugh

Keyword(s):

Oil Recovery ◽

Large Scale ◽

Probabilistic Methods ◽

Probabilistic Forecasting ◽

Resource Base ◽

Heterogeneous Reservoir ◽

Expected Performance ◽

Probabilistic Forecasts ◽

Facilities Design ◽

Early Decision

Summary The First-Eocene heavy-oil reservoir (1E) in the Wafra field is a candidate for steamflooding because of its world-class resource base and low-estimated primary recovery. However, industry has little experience in steamflooding carbonate reservoirs, which has prompted the staging of several 1E steamflooding tests, the latest of which is the large-scale pilot (LSP) started in 2009. To assist in facilities design, to help understand expected performance in a very heterogeneous reservoir, and to provide input to early-decision analyses, numerical thermal simulation was used to generate probabilistic forecasts. When adequate pilot history was available, the model was validated with probabilistic methods. The LSP model contained 1.5 million cells, which allowed the maintenance of adequate resolution and proper boundary conditions in the pilot area. Parallel computation enabled a probabilistic workflow to be implemented with this large thermal model. In this paper, we highlight the methodologies and inputs used to generate the probabilistic forecasts and validate the model. Major results of this work include the following: In contrast to many greenfield forecasts, the LSP forecasts were conservative, likely because of the unique aspects of the forecasting methodology, proper selection of uncertainty ranges, and the relatively high density of input data for model construction; wide variations in production metrics were forecast, indicative of a highly heterogeneous reservoir; results indicated that the validated model adequately captured the global or statistical pilot heterogeneity, enabling proper capture of steamflood flow/drainage mechanisms; and despite this heterogeneity, forecast oil-recovery levels were comparable with those observed in steamfloods in sandstone reservoirs.

Download Full-text

Questionable Research Practices in Ecology and Evolution

10.31219/osf.io/ajyqg ◽

2018 ◽

Author(s):

Hannah Fraser ◽

Timothy H. Parker ◽

Shinichi Nakagawa ◽

Ashley Barnett ◽

Fiona Fidler

Keyword(s):

Large Scale ◽

Research Community ◽

Unexpected Finding ◽

Research Practices ◽

Questionable Research Practices ◽

Replication Studies ◽

Cherry Picking

We surveyed 807 researchers (494 ecologists and 313 evolutionary biologists) about their use of Questionable Research Practices (QRPs), including cherry picking statistically significant results, p hacking, and hypothesising after the results are known (HARKing). We also asked them to estimate the proportion of their colleagues that use each of these QRPs. Several of the QRPs were prevalent within the ecology and evolution research community. Across the two groups, we found 64% of surveyed researchers reported they had at least once failed to report results because they were not statistically significant (cherry picking); 42% had collected more data after inspecting whether results were statistically significant (a form of p hacking) and 51% had reported an unexpected finding as though it had been hypothesised from the start (HARKing). Such practices have been directly implicated in the low rates of reproducible results uncovered by recent large scale replication studies in psychology and other disciplines. The rates of QRPs found in this study are comparable with the rates seen in psychology, indicating that the reproducibility problems discovered in psychology are also likely to be present in ecology and evolution.

Download Full-text

Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery

Statistical Methods in Medical Research ◽

10.1177/0962280211403604 ◽

2011 ◽

Vol 22 (1) ◽

pp. 57-69 ◽

Cited By ~ 62

Author(s):

G Niklas Norén ◽

Johan Hopstadius ◽

Andrew Bate

Keyword(s):

Observational Data ◽

Large Scale ◽

Statistical Significance ◽

Pattern Discovery ◽

Null Model ◽

Improve Patient Care ◽

Data Sets ◽

Practical Relevance ◽

Random Fluctuations ◽

Event Rates

Large observational data sets are a great asset to better understand the effects of medicines in clinical practice and, ultimately, improve patient care. For an empirical pattern in observational data to be of practical relevance, it should represent a substantial deviation from the null model. For the purpose of identifying such deviations, statistical significance tests are inadequate, as they do not on their own distinguish the magnitude of an effect from its data support. The observed-to-expected (OE) ratio on the other hand directly measures strength of association and is an intuitive basis to identify a range of patterns related to event rates, including pairwise associations, higher order interactions and temporal associations between events over time. It is sensitive to random fluctuations for rare events with low expected counts but statistical shrinkage can protect against spurious associations. Shrinkage OE ratios provide a simple but powerful framework for large-scale pattern discovery. In this article, we outline a range of patterns that are naturally viewed in terms of OE ratios and propose a straightforward and effective statistical shrinkage transformation that can be applied to any such ratio. The proposed approach retains emphasis on the practical relevance and transparency of highlighted patterns, while protecting against spurious associations.

Download Full-text

Questionable Research Practices in Ecology and Evolution

10.32942/osf.io/ajyqg ◽

2018 ◽

Author(s):

Hannah Fraser ◽

Timothy H. Parker ◽

Shinichi Nakagawa ◽

Ashley Barnett ◽

Fiona Fidler

Keyword(s):

Large Scale ◽

Research Community ◽

Unexpected Finding ◽

Research Practices ◽

Questionable Research Practices ◽

Replication Studies ◽

Cherry Picking

Download Full-text

Analysis of the Temporal and Spatial Distribution Patterns of Abnormal Vertical Growth in Commercial Macadamia Orchards in Australia

Proceedings ◽

10.3390/proceedings2019036047 ◽

2020 ◽

Vol 36 (1) ◽

pp. 47

Author(s):

Zakeel ◽

Geering ◽

Akinsanmi

Keyword(s):

Binary Data ◽

Large Scale ◽

Goodness Of Fit ◽

Temporal Dynamics ◽

Statistical Significance ◽

Distribution Patterns ◽

R Package ◽

Point Pattern ◽

Temporal And Spatial Distribution ◽

Vertical Growth

Abnormal vertical growth (AVG) syndrome, which has an unknown aetiology, is a serious threat to the Australian macadamia industry. AVG is characterized by vigorous upright growth and reduced flowering and nut set that results in over 70% yield loss. However, there is a deficiency in knowledge about the distribution of AVG. In this study, we used spatial analysis to provide insights into the distribution and spread of AVG in commercial macadamia orchards in Australia. Using binary data of AVG occurrence from large-scale surveys of six affected commercial orchards in Queensland (five orchards) and New South Wales (one orchard) in 2012 and 2018, spatio-temporal dynamics of AVG was evaluated. Data were subjected to point-pattern and geostatistical analyses using the R package EPIPHY. The Fisher’s index of dispersion of all orchards showed aggregated patterns of affected trees in both years, with statistical significance (p < 0.01) of chi-square test. Goodness-of-fit comparisons of incidence data of all orchards with β-binomial distributions showed that AVG incidence increased by 64% over the six-year period. AVG distribution and the β-binomial parameters exhibited strong heterogeneity, which indicates high degree of aggregation and increasing spread of AVG over time. In addition, binary power law and spatial hierarchy tests confirmed the patterns of aggregation in all orchards. These results implicate a biotic agent as the cause of AVG.

Download Full-text

Inferring protein sequence-function relationships with large-scale positive-unlabeled learning

10.1101/2020.08.19.257642 ◽

2020 ◽

Cited By ~ 2

Author(s):

Hyebin Song ◽

Bennett J. Bremer ◽

Emily C. Hinds ◽

Garvesh Raskutti ◽

Philip A. Romero

Keyword(s):

Protein Sequence ◽

Large Scale ◽

Sampling Error ◽

Predictive Performance ◽

Data Sets ◽

Learning Framework ◽

Estimated Parameters ◽

Pu Learning ◽

And Function ◽

Key Residues

SummaryMachine learning can infer how protein sequence maps to function without requiring a detailed understanding of the underlying physical or biological mechanisms. It’s challenging to apply existing supervised learning frameworks to large-scale experimental data generated by deep mutational scanning (DMS) and related methods. DMS data often contain high dimensional and correlated sequence variables, experimental sampling error and bias, and the presence of missing data. Importantly, most DMS data do not contain examples of negative sequences, making it challenging to directly estimate how sequence affects function. Here, we develop a positive-unlabeled (PU) learning framework to infer sequence-function relationships from large-scale DMS data. Our PU learning method displays excellent predictive performance across ten large-scale sequence-function data sets, representing proteins of different folds, functions, and library types. The estimated parameters pinpoint key residues that dictate protein structure and function. Finally, we apply our statistical sequence-function model to design highly stabilized enzymes.

Download Full-text

mixOmics: an R package for ‘omics feature selection and multiple data integration

10.1101/108597 ◽

2017 ◽

Cited By ~ 19

Author(s):

Florian Rohart ◽

Benoît Gautier ◽

Amrit Singh ◽

Kim-Anh Lê Cao

Keyword(s):

Data Integration ◽

Large Scale ◽

Relevant Information ◽

R Package ◽

Biological Data ◽

Molecular Signature ◽

Single Type ◽

Data Sets ◽

Omics Data ◽

Wide Range

AbstractThe advent of high throughput technologies has led to a wealth of publicly available ‘omics data coming from different sources, such as transcriptomics, proteomics, metabolomics. Combining such large-scale biological data sets can lead to the discovery of important biological insights, provided that relevant information can be extracted in a holistic manner. Current statistical approaches have been focusing on identifying small subsets of molecules (a ‘molecular signature’) to explain or predict biological conditions, but mainly for a single type of ‘omics. In addition, commonly used methods are univariate and consider each biological feature independently.We introducemixOmics, an R package dedicated to the multivariate analysis of biological data sets with a specific focus on data exploration, dimension reduction and visualisation. By adopting a system biology approach, the toolkit provides a wide range of methods that statistically integrate several data sets at once to probe relationships between heterogeneous ‘omics data sets. Our recent methods extend Projection to Latent Structure (PLS) models for discriminant analysis, for data integration across multiple ‘omics data or across independent studies, and for the identification of molecular signatures. We illustrate our latestmixOmicsintegrative frameworks for the multivariate analyses of ‘omics data available from the package.

Download Full-text