scholarly journals The assessment of intrinsic credibility and a new argument for p < 0.005

2019 ◽  
Vol 6 (3) ◽  
pp. 181534 ◽  
Author(s):  
Leonhard Held
Keyword(s):  
Confidence Interval ◽  
Open Science ◽  
Significant Finding ◽  
Tail Probabilities ◽  
Original Experiment ◽  
Significance Level ◽  
Psychological Science ◽  
Replication Experiment ◽  
Science Collaboration ◽  
Bayesian Prior

The concept of intrinsic credibility has been recently introduced to check the credibility of ‘out of the blue’ findings without any prior support. A significant result is deemed intrinsically credible if it is in conflict with a sceptical prior derived from the very same data that would make the effect just non-significant. In this paper, I propose to use Bayesian prior-predictive tail probabilities to assess intrinsic credibility. For the standard 5% significance level, this leads to a new p -value threshold that is remarkably close to the recently proposed p < 0.005 standard. I also introduce the credibility ratio, the ratio of the upper to the lower limit (or vice versa ) of a confidence interval for a significant effect size. I show that the credibility ratio has to be smaller than 5.8 such that a significant finding is also intrinsically credible. Finally, a p -value for intrinsic credibility is proposed that is a simple function of the ordinary p -value and has a direct frequentist interpretation in terms of the probability of replicating an effect. An application to data from the Open Science Collaboration study on the reproducibility of psychological science suggests that intrinsic credibility of the original experiment is better suited to predict the success of a replication experiment than standard significance.

scholarly journals Maximizing the Reproducibility of Your Research

2016 ◽  
Author(s):  
Frank Bosco ◽  
Joshua Carp ◽  
James G. Field ◽  
Hans IJzerman ◽  
Melissa Lewis ◽  
...  
Keyword(s):  
New York ◽  
Open Science ◽  
Psychological Science ◽  
Science Collaboration

Open Science Collaboration (in press). Maximizing the reproducibility of your research. In S. O. Lilienfeld & I. D. Waldman (Eds.), Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions. New York, NY: Wiley.

scholarly journals The replication and reproducibility crises: origins and consequences for studies of ecology and evolution

2018 ◽  
Author(s):  
Nigel Gilles Yoccoz
Keyword(s):  
Scientific Progress ◽  
Present Situation ◽  
Open Science ◽  
P Values ◽  
Psychological Science ◽  
Alternative Hypotheses ◽  
Research Findings ◽  
Fundamental Misunderstanding ◽  
Science Collaboration ◽  
Published Research

Watch the VIDEO.There is a widespread discussion around a scientific crisis, resulting from a lack of reproducibility of published scientific studies. This was exemplified by Ioannidis’ 2005 paper “Why most published research findings are false” or the 2015 Open Science Collaboration study assessing reproducibility of psychological science. An often-cited reason for this reproducibility crisis is a fundamental misunderstanding of what statistical methods, and in particular P-values, can achieve. In the context of studies of ecology and evolution, I will show how 1) the pressure for publishing “novel” results, 2) what Gelman has called the “garden of forking paths”, i.e. the fact that published analyses represent only one out of many possible analyses, and 3) the often fruitless dichotomy between a null and alternative hypotheses, has led to the present situation. While scientific progress is dependent of major breakthroughs, we also need to find a better balance between confirmatory research – understanding how known effects vary in size according to the context – and exploratory, non-incremental research – finding new effects.

scholarly journals Ramsey_open_science

2019 ◽  
Author(s):  
Richard Ramsey
Keyword(s):  
Scientific Practice ◽  
Open Science ◽  
Research Practices ◽  
Psychological Science ◽  
Low Levels ◽  
Science Collaboration

The credibility of psychological science has been questioned recently, due to low levels of reproducibility and the routine use of inadequate research practices (Chambers, 2017; Open Science Collaboration, 2015; Simmons, Nelson, &amp; Simonsohn, 2011). In response, wide-ranging reform to scientific practice has been proposed (e.g., Munafò et al., 2017), which has been dubbed a “credibility revolution” (Vazire, 2018). My aim here is to advocate why and how we should embrace such reform, and discuss the likely implications.

scholarly journals Do we know what we are doing?

2019 ◽  
Author(s):  
Andrew Piper
Keyword(s):  
Open Science ◽  
Reproductive Failure ◽  
University Of Virginia ◽  
Time And Space ◽  
Psychological Science ◽  
The Social ◽  
Science Collaboration ◽  
Scientific Results ◽  
The University

In November 2012, the newly created Open Science Collaboration published a brief article announcing a multi-year effort to "estimate the reproducibility of psychological science." The collaboration was directed by Brian Nosek of the University of Virginia and would eventually involve over 250 co-authors. According to the collaboration, reproducibility was one of, if not the single most defining feature of the social endeavor known as "science." "Other types of belief," the authors write, "depend on the authority and motivations of the source; beliefs in science do not." The ability to reproduce scientific results across time and space -- the ability to have results be independentof the individuals involved -- is what the authors argued makes science science. And yet the eventual findings of the reproducibility project showed a remarkable reproductive failure. Over half of all studies failed to indicate similar effects upon replication. The very value upon which science was supposed to be founded appeared to be an exception rather than a norm.

Bayesian belief updating after a replication experiment

2018 ◽  
Vol 41 ◽  
Author(s):  
Alex O. Holcombe ◽  
Samuel J. Gershman
Keyword(s):  
Confirmation Theory ◽  
Bayesian Confirmation Theory ◽  
Belief Updating ◽  
Original Experiment ◽  
Bayesian Confirmation ◽  
Replication Experiment

AbstractZwaan et al. and others discuss the importance of the inevitable differences between a replication experiment and the corresponding original experiment. But these discussions are not informed by a principled, quantitative framework for taking differences into account. Bayesian confirmation theory provides such a framework. It will not entirely solve the problem, but it will lead to new insights.

scholarly journals Prevalence of hepatitis B and C seropositivity in pregnant women

2014 ◽  
Vol 48 (1) ◽  
pp. 89-96 ◽  
Author(s):  
Carla Natalina da Silva Fernandes ◽  
Michelly de Melo Alves ◽  
Michelly Lorrane de Souza ◽  
Gleyce Alves Machado ◽  
Gleiber Couto ◽  
...  
Keyword(s):  
Data Analysis ◽  
Hepatitis B ◽  
Confidence Interval ◽  
Pregnant Women ◽  
Exploratory Study ◽  
Maternity Hospital ◽  
Hospital Records ◽  
Significance Level

The aim of the study was to identify the prevalence of hepatitis B and C seropositivity in pregnant women attended in a public maternity hospital located in Catalao-GO from 2005 to 2009. Descriptive, exploratory study conducted through patients` hospital records. For data analysis, we used SPSS version 18.0. The confidence interval (CI) was calculated using the Person χ² test, considering a significance level of 5% (p <0.05). The prevalence of HBV was 5.64% and HCV 0.098%, predominantly in young pregnant women aged between 20 and 30 years old, single and in their first pregnancy.

scholarly journals Integrating open science in the teaching of cognitive research methods

2020 ◽  
Author(s):  
Ralph S. Redden ◽  
Colin R McCormick
Keyword(s):  
Research Methods ◽  
Undergraduate Research ◽  
Scientific Practice ◽  
A Priori ◽  
Daily Practice ◽  
Open Science ◽  
Methods Course ◽  
Replication Experiment ◽  
Successful Replication ◽  
Oral Presentations

Openness, transparency, and reproducibility are widely accepted as fundamental aspects of scientific practice. However, a growing body of evidence suggests that these features are not readily adopted in the daily practice of most scientists. The Centre for Open Science has been championing efforts for systemic change in the scientific process, with newly adopted practices such as preregistration and open sharing of data and experimental materials. In an effort to inculcate these practices early in training, we have integrated several key components of open science practice into an undergraduate research methods course in the cognitive sciences. Students were divided into four research teams, each with the goal of carrying out a replication experiment related to the study of attention; specifically, temporal orienting, alertness, prior entry, and the attentional blink. Teams completed a preregistration exercise, and importantly, were encouraged to consider a priori the criteria for a successful replication. They were also required to collect and analyze data, prepare manuscripts, and disseminate their findings in poster symposia and oral presentations. All project materials can be found at https://osf.io/gxkfq/. Critical appraisal of the goals and implementation of the course are discussed.

Multiple-Study Replication Initiatives

2021 ◽  
pp. 152-172
Author(s):  
R. Barker Bausell
Keyword(s):  
Health Care ◽  
Social Science ◽  
Science Studies ◽  
Open Science ◽  
Questionable Research Practices ◽  
Science Collaboration ◽  
Scientific Results ◽  
Commercial Applications ◽  
Successful Replication ◽  
Psychological Experiments

The “mass” replications of multiple studies, some employing dozens of investigators distributed among myriad sites, is unique to the reproducibility movement. The most impressive of these initiatives was employed by the Open Science Collaboration directed by Brian Nosek, who recruited 270 investigators to participate in the replication of 100 psychological experiments via a very carefully structured, prespecified protocol that avoided questionable research practices. Just before this Herculean effort, two huge biotech firms (Amegen and Bayer Health Care) respectively conducted 53 and 67 preclinical replications of promising published studies to ascertain which results were worth pursuing for commercial applications. Amazingly, in less than a 10-year period, a number of other diverse multistudy replications were also conducted involving hundreds of effects. Among these were the three “many lab” multistudy replications based on the Open Science Model (but also designed to ascertain if potential confounders of the approach itself existed, such as differences in participant types, settings, and timing), replications of social science studies published in Science and Nature, experimental economics studies, and even self-reported replications ascertained from a survey. Somewhat surprisingly, the overall successful replication percentage for this diverse collection of 811 studies was 46%, mirroring the modeling results discussed in Chapter 3 and supportive of John Ioannidis’s pejorative and often quoted conclusion that most scientific results are incorrect.

scholarly journals Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

2020 ◽  
Vol 3 (3) ◽  
pp. 309-331 ◽  
Author(s):  
Charles R. Ebersole ◽  
Maya B. Mathur ◽  
Erica Baranski ◽  
Diane-Jo Bart-Plange ◽  
Nicholas R. Buttrick ◽  
...  
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Total Sample ◽  
Median Number ◽  
Open Science ◽  
Original Study ◽  
Effect Sizes ◽  
Replication Studies ◽  
Expert Review ◽  
Science Collaboration

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

Open Science Promotes Diverse, Just, and Sustainable Research and Educational Outcomes

2019 ◽  
Vol 19 (1) ◽  
pp. 5-20 ◽  
Author(s):  
Jon E Grahe ◽  
Kelly Cuccolo ◽  
Dana C Leighton ◽  
Leslie D Cramblet Alvarez
Keyword(s):  
Educational Outcomes ◽  
Open Science ◽  
Organizational Systems ◽  
Research Projects ◽  
Equitable Distribution ◽  
The Past ◽  
Psychological Science ◽  
Collaborative Efforts

Open science initiatives, which are often collaborative efforts focused on making research more transparent, have experienced increasing popularity in the past decade. Open science principles of openness and transparency provide opportunities to advance diversity, justice, and sustainability by promoting diverse, just, and sustainable outcomes among both undergraduate and senior researchers. We review models that demonstrate the importance of greater diversity, justice, and sustainability in psychological science before describing how open science initiatives promote these values. Open science initiatives also promote diversity, justice, and sustainability through increased levels of inclusion and access, equitable distribution of opportunities and dissemination of knowledge, and increased sustainability stemming from increased generalizability. In order to provide an application of the concepts discussed, we offer a set of diversity, justice, and sustainability lens questions for individuals to use while assessing research projects and other organizational systems and consider concrete classroom applications for these initiatives.

Sign in / Sign up

Export Citation Format

Share Document