scholarly journals Research practices and statistical reporting quality in 250 economic psychology master's theses: a meta-research investigation

2019 ◽  
Vol 6 (12) ◽  
pp. 190738 ◽  
Author(s):  
Jerome Olsen ◽  
Johanna Mosen ◽  
Martin Voracek ◽  
Erich Kirchler
Keyword(s):  
Statistical Power ◽  
Testing Effect ◽  
Reporting Quality ◽  
Open Science ◽  
Sample Size Determination ◽  
Effect Sizes ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Standardized Effect Sizes ◽  
Research Findings

The replicability of research findings has recently been disputed across multiple scientific disciplines. In constructive reaction, the research culture in psychology is facing fundamental changes, but investigations of research practices that led to these improvements have almost exclusively focused on academic researchers. By contrast, we investigated the statistical reporting quality and selected indicators of questionable research practices (QRPs) in psychology students' master's theses. In a total of 250 theses, we investigated utilization and magnitude of standardized effect sizes, along with statistical power, the consistency and completeness of reported results, and possible indications of p -hacking and further testing. Effect sizes were reported for 36% of focal tests (median r = 0.19), and only a single formal power analysis was reported for sample size determination (median observed power 1 − β = 0.67). Statcheck revealed inconsistent p -values in 18% of cases, while 2% led to decision errors. There were no clear indications of p -hacking or further testing. We discuss our findings in the light of promoting open science standards in teaching and student supervision.

False-Positive Results and a Nontechnical Overview of Their Modeling

2021 ◽  
pp. 39-55
Author(s):  
R. Barker Bausell
Keyword(s):  
False Positive ◽  
Statistical Power ◽  
Research Literature ◽  
Effect Sizes ◽  
P Value ◽  
Research Practices ◽  
P Values ◽  
Questionable Research Practices ◽  
Larger Sample ◽  
Positive Results

This chapter explores three empirical concepts (the p-value, the effect size, and statistical power) integral to the avoidance of false positive scientific. Their relationship to reproducibility is explained in a nontechnical manner without formulas or statistical jargon, with p-values and statistical power presented in terms of probabilities from zero to 1.0 with the values of most interest to scientists being 0.05 (synonymous with a positive, hence, publishable result) and 0.80 (the most commonly recommended probability that a positive result will be obtained if the hypothesis that generated it was correct and the study will be properly designed and conducted). Unfortunately many scientists circumvent both by artifactually inflating the 0.05 criterion, overstating the available statistical power, and engaging in a number of other questionable research practices. These issues are discussed via statistical models from the genetic and psychological fields and then extended to a number of different p-values, statistical power levels, effect sizes, and the prevalence of “true,” effects expected to exist in the research literature. Among the basic conclusions of these modeling efforts are that employing more stringent p-values and larger sample sizes constitute the most effective statistical approaches for increasing the reproducibility of published results in all empirically based scientific literatures. This chapter thus lays the necessary foundation for understanding and appreciating the effects of appropriate p-values, sufficient statistical power, reaslistic effect sizes, and the avoidance of questionable research practices upon the production of reproducible results.

Publication bias and statistical power in gerontological psychology

2018 ◽  
Author(s):  
Christopher Brydges
Keyword(s):  
Publication Bias ◽  
Statistical Power ◽  
Research Practices ◽  
P Values ◽  
Questionable Research Practices ◽  
Gerontological Research ◽  
Article Type ◽  
Research Findings ◽  
Psychological Journal ◽  
Published Research

Objectives: Research has found evidence of publication bias, questionable research practices (QRPs), and low statistical power in published psychological journal articles. Isaacowitz’s (2018) editorial in the Journals of Gerontology Series B, Psychological Sciences called for investigation of these issues in gerontological research. The current study presents meta-research findings based on published research to explore if there is evidence of these practices in gerontological research. Method: 14,481 test statistics and p values were extracted from articles published in eight top gerontological psychology journals since 2000. Frequentist and Bayesian caliper tests were used to test for publication bias and QRPs (specifically, p-hacking and incorrect rounding of p values). A z-curve analysis was used to estimate average statistical power across studies.Results: Strong evidence of publication bias was observed, and average statistical power was approximately .70 – below the recommended .80 level. Evidence of p-hacking was mixed. Evidence of incorrect rounding of p values was inconclusive.Discussion: Gerontological research is not immune to publication bias, QRPs, and low statistical power. Researchers, journals, institutions, and funding bodies are encouraged to adopt open and transparent research practices, and using Registered Reports as an alternative article type to minimize publication bias and QRPs, and increase statistical power.

scholarly journals Open Science, Replicability, and Transparency in Modelling

2021 ◽  
pp. 175-183
Author(s):  
Toby Prike
Keyword(s):  
Computational Modelling ◽  
Open Science ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Study Materials ◽  
Research Findings ◽  
Research Questions ◽  
Study Designs ◽  
Good Research ◽  
Science Movement

AbstractRecent years have seen large changes to research practices within psychology and a variety of other empirical fields in response to the discovery (or rediscovery) of the pervasiveness and potential impact of questionable research practices, coupled with well-publicised failures to replicate published findings. In response to this, and as part of a broader open science movement, a variety of changes to research practice have started to be implemented, such as publicly sharing data, analysis code, and study materials, as well as the preregistration of research questions, study designs, and analysis plans. This chapter outlines the relevance and applicability of these issues to computational modelling, highlighting the importance of good research practices for modelling endeavours, as well as the potential of provenance modelling standards, such as PROV, to help discover and minimise the extent to which modelling is impacted by unreliable research findings from other disciplines.

Pre-Registration in the Undergraduate Dissertation: A Critical Discussion

2020 ◽  
Author(s):  
Madeleine Pownall
Keyword(s):  
Resource Constraints ◽  
Open Science ◽  
Critical Discussion ◽  
Perceived Barriers ◽  
Science Literature ◽  
Statistics Anxiety ◽  
Questionable Research Practices ◽  
Research Findings ◽  
Potential Benefits ◽  
Data Analysis Plan

Currently under review at Psychology Teaching Review. Over recent years, Psychology has become increasingly concerned with reproducibility and replicability of research findings (Munafò et al., 2017). One method of ensuring that research is hypothesis driven, as opposed to data driven, is the process of publicly pre-registering a study’s hypotheses, data analysis plan, and procedure prior to data collection (Nosek, Ebersole, DeHaven, & Mellor, 2018). This paper discusses the potential benefits of introducing pre-registration to the undergraduate dissertation. The utility of pre-registration as a pedagogic practice within dissertation supervision is also critically appraised, with reference to open science literature. Here, it is proposed that encouraging pre-registration of undergraduate dissertation work may alleviate some pedagogic challenges, such as statistics anxiety, questionable research practices, and research clarity and structure. Perceived barriers, such as time and resource constraints, are also discussed.

scholarly journals Excess success in “Don’t count calorie labeling out: Calorie counts on the left side of menu items lead to lower calorie food choices”

2019 ◽  
Author(s):  
Gregory Francis ◽  
Evelina Thunell
Keyword(s):  
Success Rate ◽  
Publication Bias ◽  
Food Choices ◽  
Effect Sizes ◽  
Sample Sizes ◽  
Research Practices ◽  
Policy Makers ◽  
Questionable Research Practices ◽  
Calorie Labeling

Based on findings from six experiments, Dallas, Liu & Ubel (2019) concluded that placing calorie labels to the left of menu items influences consumers to choose lower calorie food options. Contrary to previously reported findings, they suggested that calorie labels do influence food choices, but only when placed to the left because they are in this case read first. If true, these findings have important implications for the design of menus and may help address the obesity pandemic. However, an analysis of the reported results indicates that they seem too good to be true. We show that if the effect sizes in Dallas et al. (2019) are representative of the populations, a replication of the six studies (with the same sample sizes) has a probability of only 0.014 of producing uniformly significant outcomes. Such a low success rate suggests that the original findings might be the result of questionable research practices or publication bias. We therefore caution readers and policy makers to be skeptical about the results and conclusions reported by Dallas et al. (2019).

scholarly journals The effect of preregistration on trust in empirical research findings: results of a registered report

2020 ◽  
Vol 7 (4) ◽  
pp. 181351 ◽  
Author(s):  
Sarahanne M. Field ◽  
E.-J. Wagenmakers ◽  
Henk A. L. Kiers ◽  
Rink Hoekstra ◽  
Anja F. Ernst ◽  
...  
Keyword(s):  
Pilot Study ◽  
Empirical Evidence ◽  
Empirical Research ◽  
Strong Evidence ◽  
Bayes Factors ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Research Findings ◽  
Research Questions ◽  
Research Initiatives

The crisis of confidence has undermined the trust that researchers place in the findings of their peers. In order to increase trust in research, initiatives such as preregistration have been suggested, which aim to prevent various questionable research practices. As it stands, however, no empirical evidence exists that preregistration does increase perceptions of trust. The picture may be complicated by a researcher's familiarity with the author of the study, regardless of the preregistration status of the research. This registered report presents an empirical assessment of the extent to which preregistration increases the trust of 209 active academics in the reported outcomes, and how familiarity with another researcher influences that trust. Contrary to our expectations, we report ambiguous Bayes factors and conclude that we do not have strong evidence towards answering our research questions. Our findings are presented along with evidence that our manipulations were ineffective for many participants, leading to the exclusion of 68% of complete datasets, and an underpowered design as a consequence. We discuss other limitations and confounds which may explain why the findings of the study deviate from a previously conducted pilot study. We reflect on the benefits of using the registered report submission format in light of our results. The OSF page for this registered report and its pilot can be found here: http://dx.doi.org/10.17605/OSF.IO/B3K75 .

Collaboration in Giftedness and Talent Development Research

2020 ◽  
Vol 43 (2) ◽  
pp. 91-107
Author(s):  
Matthew C. Makel ◽  
Kendal N. Smith ◽  
Erin M. Miller ◽  
Scott J. Peters ◽  
Matthew T. McBee
Keyword(s):  
Gifted Education ◽  
Collaborative Research ◽  
Large Scale ◽  
Open Science ◽  
Accurate Information ◽  
Development Research ◽  
Research Practices ◽  
Science Practices ◽  
Research Findings ◽  
Potential Improvement

Existing research practices in gifted education have many areas for potential improvement so that they can provide useful, generalizable evidence to various stakeholders. In this article, we first review the field’s current research practices and consider the quality and utility of its research findings. Next, we discuss how open science practices increase the transparency of research so readers can more effectively evaluate its validity. Third, we introduce five large-scale collaborative research models that are being used in other fields and discuss how they could be implemented in gifted education research. Finally, we review potential challenges and limitations to implementing collaborative research models in gifted education. We believe greater use of large-scale collaboration will help the field overcome some of its methodological challenges to help provide more precise and accurate information about gifted education.

scholarly journals Calibrate your confidence in research findings: A tutorial on improving research methods and practices

2020 ◽  
Vol 14 ◽  
Author(s):  
Aline da Silva Frost ◽  
Alison Ledgerwood
Keyword(s):  
Research Methods ◽  
Statistical Power ◽  
Type I Error ◽  
Effect Sizes ◽  
Online Media ◽  
Type I ◽  
Journal Articles ◽  
Psychological Science ◽  
Different Types ◽  
Research Findings

Abstract This article provides an accessible tutorial with concrete guidance for how to start improving research methods and practices in your lab. Following recent calls to improve research methods and practices within and beyond the borders of psychological science, resources have proliferated across book chapters, journal articles, and online media. Many researchers are interested in learning more about cutting-edge methods and practices but are unsure where to begin. In this tutorial, we describe specific tools that help researchers calibrate their confidence in a given set of findings. In Part I, we describe strategies for assessing the likely statistical power of a study, including when and how to conduct different types of power calculations, how to estimate effect sizes, and how to think about power for detecting interactions. In Part II, we provide strategies for assessing the likely type I error rate of a study, including distinguishing clearly between data-independent (“confirmatory”) and data-dependent (“exploratory”) analyses and thinking carefully about different forms and functions of preregistration.

scholarly journals The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials

2021 ◽  
Author(s):  
Robert Schulz ◽  
Georg Langen ◽  
Robert Prill ◽  
Michael Cassel ◽  
Tracey Weissgerber
Keyword(s):  
Clinical Trial ◽  
Clinical Trials ◽  
Sports Medicine ◽  
Open Data ◽  
Open Science ◽  
Research Practices ◽  
Scientific Rigor ◽  
Sample Size Calculations ◽  
Research Findings ◽  
The Devil

Introduction: While transparent reporting of clinical trials is essential to assess the risk of bias and translate research findings into clinical practice, earlier studies have shown that deficiencies are common. This study examined current clinical trial reporting and transparent research practices in sports medicine and orthopedics. Methods: The sample included clinical trials published in the top 25% of sports medicine and orthopedics journals over eight months. Two independent reviewers assessed pre-registration, open data and criteria related to scientific rigor, the study sample, and data analysis. Results: The sample included 163 clinical trials from 27 journals. While the majority of trials mentioned rigor criteria, essential details were often missing. Sixty percent (confidence interval [CI] 53-68%) of trials reported sample size calculations, but only 32% (CI 25-39%) justified the expected effect size. Few trials indicated the blinding status of all main stakeholders (4%; CI 1-7%). Only 18% (CI 12-24%) included information on randomization type, method, and concealed allocation. Most trials reported participants' sex/gender (95%; CI 92-98%) and information on inclusion and exclusion criteria (78%; CI 72-84%). Only 20% (CI 14-26%) of trials were pre-registered. No trials deposited data in open repositories. Conclusions: These results will aid the sports medicine and orthopedics community in developing tailored interventions to improve reporting. While authors typically mention blinding, randomization and other factors, essential details are often missing. Greater acceptance of open science practices, like pre-registration and open data, is needed. These practices have been widely encouraged, we discuss systemic interventions that may improve clinical trial reporting. Registration: https://doi.org/10.17605/OSF.IO/9648H

scholarly journals ­Questionable Research Practices and Open Science in Quantitative Criminology

2021 ◽  
Author(s):  
Jason Chin ◽  
Justin T. Pickett ◽  
Simine Vazire ◽  
Alex O. Holcombe
Keyword(s):  
Open Science ◽  
Research Practices ◽  
Questionable Research Practices
Sign in / Sign up

Export Citation Format

Share Document