scholarly journals Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication

2017 ◽  
Author(s):  
Robbie Cornelis Maria van Aert ◽  
Marcel A. L. M. van Assen
Keyword(s):  
Hybrid Method ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Open Science ◽  
Original Study ◽  
High Rate ◽  
Size Estimation ◽  
Effect Size Estimation ◽  
Science Collaboration

The unrealistic high rate of positive results within psychology increased the attention for replication research. Researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because of it being statistically significant and this bias is not taken into consideration in traditional meta-analysis. We developed a hybrid method that does take statistical significance of the original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its good statistical properties. Applying the hybrid method to the data of the Reproducibility Project Psychology (Open Science Collaboration, 2015) demonstrated that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than reported in the original study and that some effects may be even absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and developed a web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method.

scholarly journals Designing Studies and Evaluating Research Results: Type M and Type S Errors for Pearson Correlation Coefficient

2020 ◽  
Author(s):  
Giulia Bertoldo ◽  
Claudio Zandonella Callegher ◽  
Gianmarco Altoè
Keyword(s):  
Correlation Coefficient ◽  
Effect Size ◽  
Statistical Power ◽  
Pearson Correlation ◽  
Statistical Significance ◽  
Design Analysis ◽  
Pearson Correlation Coefficient ◽  
Size Estimation ◽  
Present Contribution ◽  
Effect Size Estimation

It is widely appreciated that many studies in psychological science suffer from low statistical power. One of the consequences of analyzing underpowered studies with thresholds of statistical significance, is a high risk of finding exaggerated effect size estimates, in the right or in the wrong direction. These inferential risks can be directly quantified in terms of Type M (magnitude) error and Type S (sign) error, which directly communicate the consequences of design choices on effect size estimation. Given a study design, Type M error is the factor by which a statistically significant effect is on average exaggerated. Type S error is the probability to find a statistically significant result in the opposite direction to the plausible one. Ideally, these errors should be considered during a prospective design analysis in the design phase of a study to determine the appropriate sample size. However, they can also be considered when evaluating studies’ results in a retrospective design analysis. In the present contribution we aim to facilitate the considerations of these errors in the research practice in psychology. For this reason we illustrate how to consider Type M and Type S errors in a design analysis using one of the most common effect size measures in psychology: Pearson correlation coefficient. We provide various examples and make the R functions freely available to enable researchers to perform design analysis for their research projects.

Assessing Evidence for Replication: A Likelihood-Based Approach

2020 ◽  
Author(s):  
Peter Dixon ◽  
Scott Glover
Keyword(s):  
Likelihood Ratio ◽  
Effect Size ◽  
Null Model ◽  
Open Science ◽  
Original Study ◽  
Experimental Methodology ◽  
Fundamental Issue ◽  
Interesting Effect ◽  
Model Based ◽  
Science Collaboration

How to evaluate replications is a fundamental issue in experimental methodology. We develop a likelihood-based approach to assessing evidence for replication. In this approach, the design of the original study is used to derive an estimate of a theoretically interesting effect size.A likelihood ratio is then calculated to contrast the match of two models to the data from the replication attempt: 1) A model based on the derived theoretically interesting effect size; and 2) a null model. This approach provides new insights not available with existing methods of assessingreplication. When applied to data from the Replication Project (Open Science Collaboration, 2015), the procedure indicates that a large portion of the replications failed to find evidence for a theoretically interesting effect.

scholarly journals Potential Therapeutic Effects of Curcumin on Glycemic and Lipid Profile in Uncomplicated Type 2 Diabetes—A Meta-Analysis of Randomized Controlled Trial

Nutrients ◽  
2021 ◽  
Vol 13 (2) ◽  
pp. 404
Author(s):  
Emma Altobelli ◽  
Paolo Matteo Angeletti ◽  
Ciro Marziliano ◽  
Marianna Mastrodomenico ◽  
Anna Rita Giuliani ◽  
...  
Keyword(s):  
Type 2 Diabetes ◽  
Lipid Profile ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Glycosylated Hemoglobin ◽  
Cochrane Library ◽  
Therapeutic Effects ◽  
Model Assessment

Diabetes mellitus is an important issue for public health, and it is growing in the world. In recent years, there has been a growing research interest on efficacy evidence of the curcumin use in the regulation of glycemia and lipidaemia. The molecular structure of curcumins allows to intercept reactive oxygen species (ROI) that are particularly harmful in chronic inflammation and tumorigenesis models. The aim of our study performed a systematic review and meta-analysis to evaluate the effect of curcumin on glycemic and lipid profile in subjects with uncomplicated type 2 diabetes. The papers included in the meta-analysis were sought in the MEDLINE, EMBASE, Scopus, Clinicaltrials.gov, Web of Science, and Cochrane Library databases as of October 2020. The sizes were pooled across studies in order to obtain an overall effect size. A random effects model was used to account for different sources of variation among studies. Cohen’s d, with 95% confidence interval (CI) was used as a measure of the effect size. Heterogeneity was assessed while using Q statistics. The ANOVA-Q test was used to value the differences among groups. Publication bias was analyzed and represented by a funnel plot. Curcumin treatment does not show a statistically significant reduction between treated and untreated patients. On the other hand, glycosylated hemoglobin, homeostasis model assessment (HOMA), and low-density lipoprotein (LDL) showed a statistically significant reduction in subjects that were treated with curcumin, respectively (p = 0.008, p < 0.001, p = 0.021). When considering HBA1c, the meta-regressions only showed statistical significance for gender (p = 0.034). Our meta-analysis seems to confirm the benefits on glucose metabolism, with results that appear to be more solid than those of lipid metabolism. However, further studies are needed in order to test the efficacy and safety of curcumin in uncomplicated type 2 diabetes.

Do sexist jokes increase rape proclivity among males high in hostile sexism? Evidence from two pre-registered direct replications of Thomae & Viki (2013)

2021 ◽  
Author(s):  
Neil McLatchie ◽  
Manuela Thomae
Keyword(s):  
Bayes Factor ◽  
Alternative Hypothesis ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Original Study ◽  
Bayes Factors ◽  
Hostile Sexism ◽  
Replication Studies ◽  
Open Research ◽  
Rape Proclivity

Thomae and Viki (2013) reported that increased exposure to sexist humour can increase rape proclivity among males, specifically those who score high on measures of Hostile Sexism. Here we report two pre-registered direct replications (N = 530) of Study 2 from Thomae and Viki (2013) and assess replicability via (i) statistical significance, (ii) Bayes factors, (iii) the small-telescope approach, and (iv) an internal meta-analysis across the original and replication studies. The original results were not supported by any of the approaches. Combining the original study and the replications yielded moderate evidence in support of the null over the alternative hypothesis with a Bayes factor of B = 0.13. In light of the combined evidence, we encourage researchers to exercise caution before claiming that brief exposure to sexist humour increases male’s proclivity towards rape, until further pre-registered and open research demonstrates the effect is reliably reproducible.

Sign in / Sign up

Export Citation Format

Share Document