scholarly journals Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

2020 ◽  
Vol 3 (3) ◽  
pp. 309-331 ◽  
Author(s):  
Charles R. Ebersole ◽  
Maya B. Mathur ◽  
Erica Baranski ◽  
Diane-Jo Bart-Plange ◽  
Nicholas R. Buttrick ◽  
...  
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Total Sample ◽  
Median Number ◽  
Open Science ◽  
Original Study ◽  
Effect Sizes ◽  
Replication Studies ◽  
Expert Review ◽  
Science Collaboration

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

scholarly journals Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability

2019 ◽  
Author(s):  
Charles R. Ebersole ◽  
Maya B Mathur ◽  
Erica Baranski ◽  
Diane-Jo Bart-Plange ◽  
Nick Buttrick ◽  
...  
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Total Sample ◽  
Median Number ◽  
Open Science ◽  
Original Study ◽  
Effect Sizes ◽  
Expert Review ◽  
Science Collaboration ◽  
Cumulative Evidence

Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p &lt; .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised protocols in multiple laboratories (Median number of laboratories per original study = 6.5; Range 3 to 9; Median total sample = 1279.5; Range 276 to 3512) for high-powered tests of each original finding with both protocols. Overall, Revised protocols produced similar effect sizes as RP:P protocols following the preregistered analysis plan (Δr = .002 or .014, depending on analytic approach). The median effect size for Revised protocols (r = .05) was similar to RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than the original studies (r = .37). The cumulative evidence of original study and three replication attempts suggests that effect sizes for all 10 (median r = .07; range .00 to .15) are 78% smaller on average than original findings (median r = .37; range .19 to .50), with very precisely estimated effects.

scholarly journals Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability

2020 ◽  
Author(s):  
Charles R. Ebersole ◽  
Brian A. Nosek ◽  
Mallory Kidwell ◽  
Nick Buttrick ◽  
Erica Baranski ◽  
...  
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Total Sample ◽  
Median Number ◽  
Open Science ◽  
Original Study ◽  
Effect Sizes ◽  
Expert Review ◽  
Science Collaboration ◽  
Cumulative Evidence

Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p &lt; .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised protocols in multiple laboratories (Median number of laboratories per original study = 6.5; Range 3 to 9; Median total sample = 1279.5; Range 276 to 3512) for high-powered tests of each original finding with both protocols. Overall, Revised protocols produced similar effect sizes as RP:P protocols following the preregistered analysis plan (Δr = .002 or .014, depending on analytic approach). The median effect size for Revised protocols (r = .05) was similar to RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than the original studies (r = .37). The cumulative evidence of original study and three replication attempts suggests that effect sizes for all 10 (median r = .07; range .00 to .15) are 78% smaller on average than original findings (median r = .37; range .19 to .50), with very precisely estimated effects.

scholarly journals Bayesian evaluation of replication studies

2020 ◽  
Author(s):  
Hidde Jelmer Leplaa ◽  
Charlotte Rietbergen ◽  
Herbert Hoijtink
Keyword(s):  
Bayes Factor ◽  
Open Science ◽  
Original Study ◽  
Replication Study ◽  
Research Results ◽  
Replication Studies ◽  
Science Collaboration ◽  
Using Data

In this paper a method is proposed to determine whether the result from an original study is corroborated in a replication study. The paper is illustrated using data from the reproducibility project psychology by the Open Science Collaboration. This method emphasizes the need to determine what one wants to replicate: the hypotheses as formulated in the introduction of the original paper, or hypotheses derived from the research results presented in the original paper. The Bayes factor will be used to determine whether the hypotheses evaluated in/resulting from the original study are corroborated by the replication study. Our method to assess the successfulness of replication will better fit the needs and desires of researchers in fields that use replication studies.

Assessing Evidence for Replication: A Likelihood-Based Approach

2020 ◽  
Author(s):  
Peter Dixon ◽  
Scott Glover
Keyword(s):  
Likelihood Ratio ◽  
Effect Size ◽  
Null Model ◽  
Open Science ◽  
Original Study ◽  
Experimental Methodology ◽  
Fundamental Issue ◽  
Interesting Effect ◽  
Model Based ◽  
Science Collaboration

How to evaluate replications is a fundamental issue in experimental methodology. We develop a likelihood-based approach to assessing evidence for replication. In this approach, the design of the original study is used to derive an estimate of a theoretically interesting effect size.A likelihood ratio is then calculated to contrast the match of two models to the data from the replication attempt: 1) A model based on the derived theoretically interesting effect size; and 2) a null model. This approach provides new insights not available with existing methods of assessingreplication. When applied to data from the Replication Project (Open Science Collaboration, 2015), the procedure indicates that a large portion of the replications failed to find evidence for a theoretically interesting effect.

scholarly journals OpenDRS: An Open-source 24-hour Recall for Mobile Devices (P13-004-19)

2019 ◽  
Vol 3 (Supplement_1) ◽  
Author(s):  
Bess Caswell ◽  
Charles Arnold ◽  
Jennie Davis ◽  
Jody Miller ◽  
Reina Engle-Stone ◽  
...  
Keyword(s):  
Data Collection ◽  
Open Source ◽  
Median Number ◽  
Open Science ◽  
Portion Size ◽  
Size Estimation ◽  
Nutrition Survey ◽  
Response Options ◽  
Electronic Survey ◽  
24 Hour Recall

Abstract Objectives Our aim was to develop a 24-hour recall program using open-source language that can be readily adapted for use on multiple platforms and in different settings. Methods We developed the Open Dietary Recall System (OpenDRS), a multi-pass, 24-hour recall survey using the XLSForm programming language. XLSForm operates in Microsoft Excel and is used by several free or subscription-based electronic survey platforms that deploy forms via Android devices or online. The first pass collects a brief list of foods consumed, the second pass collects descriptive details and added ingredients for each food, and the third pass collects portion size estimates. The final pass reviews the recorded data with the option to edit or add foods. The program references external food and ingredient lists in .csv format. These food and ingredient lists are customizable for each survey setting, can be coded to match nutrient composition or recipe tables, and can be amended over the course of a study. Questions or response options can be added or edited to fit study-specific data collection methods. Photos can be incorporated for food identification or portion size estimation. Two case studies from adaptations in Malawi and Haiti will be presented. Results OpenDRS was used to collect 24-hour dietary intake data among 6- to 15-month-old children in Malawi, for a randomized, controlled complementary feeding trial. The median number of foods reported was 6, and the median survey duration was 9 minutes. For a national nutrition survey of 6- to 59-month-old children and their caregivers in Haiti, the program was expanded to record recipe data and to capture caregiver and child intakes in one electronic survey form. Recipes reported during the caregiver's recall can be linked to the child's recall if both consumed the same mixed dish. Conclusions OpenDRS is an open-source 24-hour recall program which has been used in low- and middle-income countries. The XLSForm program for data collection and Stata code for data formatting will be made available to researchers conducting nutrition surveys via the Open Science Framework. Funding Sources The Bill & Melinda Gates Foundation and GAIN.

scholarly journals Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication

2017 ◽  
Author(s):  
Robbie Cornelis Maria van Aert ◽  
Marcel A. L. M. van Assen
Keyword(s):  
Hybrid Method ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Open Science ◽  
Original Study ◽  
High Rate ◽  
Size Estimation ◽  
Effect Size Estimation ◽  
Science Collaboration

The unrealistic high rate of positive results within psychology increased the attention for replication research. Researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because of it being statistically significant and this bias is not taken into consideration in traditional meta-analysis. We developed a hybrid method that does take statistical significance of the original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its good statistical properties. Applying the hybrid method to the data of the Reproducibility Project Psychology (Open Science Collaboration, 2015) demonstrated that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than reported in the original study and that some effects may be even absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and developed a web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method.

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews &amp; Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

MOTIVASI KERJA PERAWAT DI RUANG RAWAT INAP RSUD MAJALAYA KABUPATEN BANDUNG

2020 ◽  
Author(s):  
Andria Pragholapati
Keyword(s):  
Data Collection ◽  
Work Environment ◽  
Sampling Method ◽  
Work Motivation ◽  
Survey Methods ◽  
Sampling Technique ◽  
Total Sample ◽  
Regional Hospital ◽  
High Work

Work motivation is an influential condition for arousing, directing, and maintaining behavior related to the work environment including nurse work motivation. The purpose of this study was to edit the Nurses' Work Motivation in the Inpatient Room of Majalaya Regional Hospital. This type of research uses analytic survey methods. The sampling method uses a total sampling technique with a total sample of 55 nurses in 6 inpatients. Data collection techniques using a work motivation questionnaire. The analysis used is univariate. The results of the study 28 people (50.9%) have high work motivation. The conclusion of the results of this study some nurses have work motivation of nurses in the inpatient room of Majalaya Regional Hospital. Based on the results of the study are expected to require motivation support to increase work motivation of nurses.

Pengaruh Konsumsi Cookies Kerang Dara (Anadara Granosa) terhadap Perubahan Kadar Haemoglobin Wanita Usia Subur

2020 ◽  
Vol 7 (2) ◽  
pp. 169-179 ◽  
Author(s):  
Vera Iriani Abdullah ◽  
C.H Haumahu
Keyword(s):  
Data Collection ◽  
Perinatal Death ◽  
Design Method ◽  
Statistical Tests ◽  
Total Sample ◽  
Control Group ◽  
Childbearing Age ◽  
Control Group Design ◽  
Quasi Experimental ◽  
Anadara Granosa

In the world, around 1.62 billion people have low HB levels, around 30.2% occur in the group of women aged 15-49. In Indonesia, anemia cases rank 4th in the top 10 most disease groups. Prolonged iron deficiency can cause anemia, so it needs to be treated immediately so it doesn’t continue into pregnancy age which can cause complications until maternal and perinatal death. Papua has an abundance of marine wealth; one of those is Kerang Dara. Through this study, researchers wanted to explore the health benefits of nature. The aim is to see the effect of Consumption of Kerang Dara Cookies (Anadara Granosa) on Changes of Hemoglobin Levels of in Woman of Childbearing Age as an Effort to Prevent Anemia in District Aimas, Klaigit Village. Type of this research is quasi-experimental by pretest-posttest control group design method. The population of all the women of Childbearing who live in Klaigit Village, with a total sample of 14 people divided into 2 groups: control and intervention. Data collection techniques using random sampling. The time of data collection occurred for 2 weeks, starting from September 20th to October 4th. The results based on statistical tests using T-Test, then the value 884 is greater than the value of the table 0.05, then the conclusion is there is an Effect of Consumption Kerang Dara Cookies (Anadara Granosa) Towards the Increased of Woman of Childbearing Hemoglobin Levels in Klaigit Village in District Aimas in 2019.

scholarly journals Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece

Publications ◽  
2021 ◽  
Vol 9 (2) ◽  
pp. 14
Author(s):  
Eirini Delikoura ◽  
Dimitrios Kouis
Keyword(s):  
Peer Review ◽  
Evaluation System ◽  
Early Stage ◽  
Open Science ◽  
Health Sciences ◽  
Research Data ◽  
Stage Of Development ◽  
Open Research ◽  
Proper Training ◽  
To Receive

Recently significant initiatives have been launched for the dissemination of Open Access as part of the Open Science movement. Nevertheless, two other major pillars of Open Science such as Open Research Data (ORD) and Open Peer Review (OPR) are still in an early stage of development among the communities of researchers and stakeholders. The present study sought to unveil the perceptions of a medical and health sciences community about these issues. Through the investigation of researchers` attitudes, valuable conclusions can be drawn, especially in the field of medicine and health sciences, where an explosive growth of scientific publishing exists. A quantitative survey was conducted based on a structured questionnaire, with 179 valid responses. The participants in the survey agreed with the Open Peer Review principles. However, they ignored basic terms like FAIR (Findable, Accessible, Interoperable, and Reusable) and appeared incentivized to permit the exploitation of their data. Regarding Open Peer Review (OPR), participants expressed their agreement, implying their support for a trustworthy evaluation system. Conclusively, researchers need to receive proper training for both Open Research Data principles and Open Peer Review processes which combined with a reformed evaluation system will enable them to take full advantage of the opportunities that arise from the new scholarly publishing and communication landscape.

Sign in / Sign up

Export Citation Format

Share Document