scholarly journals Open Science, Replicability, and Transparency in Modelling

2021 ◽  
pp. 175-183
Author(s):  
Toby Prike
Keyword(s):  
Computational Modelling ◽  
Open Science ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Study Materials ◽  
Research Findings ◽  
Research Questions ◽  
Study Designs ◽  
Good Research ◽  
Science Movement

AbstractRecent years have seen large changes to research practices within psychology and a variety of other empirical fields in response to the discovery (or rediscovery) of the pervasiveness and potential impact of questionable research practices, coupled with well-publicised failures to replicate published findings. In response to this, and as part of a broader open science movement, a variety of changes to research practice have started to be implemented, such as publicly sharing data, analysis code, and study materials, as well as the preregistration of research questions, study designs, and analysis plans. This chapter outlines the relevance and applicability of these issues to computational modelling, highlighting the importance of good research practices for modelling endeavours, as well as the potential of provenance modelling standards, such as PROV, to help discover and minimise the extent to which modelling is impacted by unreliable research findings from other disciplines.

scholarly journals The effect of preregistration on trust in empirical research findings: results of a registered report

2020 ◽  
Vol 7 (4) ◽  
pp. 181351 ◽  
Author(s):  
Sarahanne M. Field ◽  
E.-J. Wagenmakers ◽  
Henk A. L. Kiers ◽  
Rink Hoekstra ◽  
Anja F. Ernst ◽  
...  
Keyword(s):  
Pilot Study ◽  
Empirical Evidence ◽  
Empirical Research ◽  
Strong Evidence ◽  
Bayes Factors ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Research Findings ◽  
Research Questions ◽  
Research Initiatives

The crisis of confidence has undermined the trust that researchers place in the findings of their peers. In order to increase trust in research, initiatives such as preregistration have been suggested, which aim to prevent various questionable research practices. As it stands, however, no empirical evidence exists that preregistration does increase perceptions of trust. The picture may be complicated by a researcher's familiarity with the author of the study, regardless of the preregistration status of the research. This registered report presents an empirical assessment of the extent to which preregistration increases the trust of 209 active academics in the reported outcomes, and how familiarity with another researcher influences that trust. Contrary to our expectations, we report ambiguous Bayes factors and conclude that we do not have strong evidence towards answering our research questions. Our findings are presented along with evidence that our manipulations were ineffective for many participants, leading to the exclusion of 68% of complete datasets, and an underpowered design as a consequence. We discuss other limitations and confounds which may explain why the findings of the study deviate from a previously conducted pilot study. We reflect on the benefits of using the registered report submission format in light of our results. The OSF page for this registered report and its pilot can be found here: http://dx.doi.org/10.17605/OSF.IO/B3K75 .

scholarly journals Research practices and statistical reporting quality in 250 economic psychology master's theses: a meta-research investigation

2019 ◽  
Vol 6 (12) ◽  
pp. 190738 ◽  
Author(s):  
Jerome Olsen ◽  
Johanna Mosen ◽  
Martin Voracek ◽  
Erich Kirchler
Keyword(s):  
Statistical Power ◽  
Testing Effect ◽  
Reporting Quality ◽  
Open Science ◽  
Sample Size Determination ◽  
Effect Sizes ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Standardized Effect Sizes ◽  
Research Findings

The replicability of research findings has recently been disputed across multiple scientific disciplines. In constructive reaction, the research culture in psychology is facing fundamental changes, but investigations of research practices that led to these improvements have almost exclusively focused on academic researchers. By contrast, we investigated the statistical reporting quality and selected indicators of questionable research practices (QRPs) in psychology students' master's theses. In a total of 250 theses, we investigated utilization and magnitude of standardized effect sizes, along with statistical power, the consistency and completeness of reported results, and possible indications of p -hacking and further testing. Effect sizes were reported for 36% of focal tests (median r = 0.19), and only a single formal power analysis was reported for sample size determination (median observed power 1 − β = 0.67). Statcheck revealed inconsistent p -values in 18% of cases, while 2% led to decision errors. There were no clear indications of p -hacking or further testing. We discuss our findings in the light of promoting open science standards in teaching and student supervision.

Pre-Registration in the Undergraduate Dissertation: A Critical Discussion

2020 ◽  
Author(s):  
Madeleine Pownall
Keyword(s):  
Resource Constraints ◽  
Open Science ◽  
Critical Discussion ◽  
Perceived Barriers ◽  
Science Literature ◽  
Statistics Anxiety ◽  
Questionable Research Practices ◽  
Research Findings ◽  
Potential Benefits ◽  
Data Analysis Plan

Currently under review at Psychology Teaching Review. Over recent years, Psychology has become increasingly concerned with reproducibility and replicability of research findings (Munafò et al., 2017). One method of ensuring that research is hypothesis driven, as opposed to data driven, is the process of publicly pre-registering a study’s hypotheses, data analysis plan, and procedure prior to data collection (Nosek, Ebersole, DeHaven, & Mellor, 2018). This paper discusses the potential benefits of introducing pre-registration to the undergraduate dissertation. The utility of pre-registration as a pedagogic practice within dissertation supervision is also critically appraised, with reference to open science literature. Here, it is proposed that encouraging pre-registration of undergraduate dissertation work may alleviate some pedagogic challenges, such as statistics anxiety, questionable research practices, and research clarity and structure. Perceived barriers, such as time and resource constraints, are also discussed.

Collaboration in Giftedness and Talent Development Research

2020 ◽  
Vol 43 (2) ◽  
pp. 91-107
Author(s):  
Matthew C. Makel ◽  
Kendal N. Smith ◽  
Erin M. Miller ◽  
Scott J. Peters ◽  
Matthew T. McBee
Keyword(s):  
Gifted Education ◽  
Collaborative Research ◽  
Large Scale ◽  
Open Science ◽  
Accurate Information ◽  
Development Research ◽  
Research Practices ◽  
Science Practices ◽  
Research Findings ◽  
Potential Improvement

Existing research practices in gifted education have many areas for potential improvement so that they can provide useful, generalizable evidence to various stakeholders. In this article, we first review the field’s current research practices and consider the quality and utility of its research findings. Next, we discuss how open science practices increase the transparency of research so readers can more effectively evaluate its validity. Third, we introduce five large-scale collaborative research models that are being used in other fields and discuss how they could be implemented in gifted education research. Finally, we review potential challenges and limitations to implementing collaborative research models in gifted education. We believe greater use of large-scale collaboration will help the field overcome some of its methodological challenges to help provide more precise and accurate information about gifted education.

scholarly journals The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials

2021 ◽  
Author(s):  
Robert Schulz ◽  
Georg Langen ◽  
Robert Prill ◽  
Michael Cassel ◽  
Tracey Weissgerber
Keyword(s):  
Clinical Trial ◽  
Clinical Trials ◽  
Sports Medicine ◽  
Open Data ◽  
Open Science ◽  
Research Practices ◽  
Scientific Rigor ◽  
Sample Size Calculations ◽  
Research Findings ◽  
The Devil

Introduction: While transparent reporting of clinical trials is essential to assess the risk of bias and translate research findings into clinical practice, earlier studies have shown that deficiencies are common. This study examined current clinical trial reporting and transparent research practices in sports medicine and orthopedics. Methods: The sample included clinical trials published in the top 25% of sports medicine and orthopedics journals over eight months. Two independent reviewers assessed pre-registration, open data and criteria related to scientific rigor, the study sample, and data analysis. Results: The sample included 163 clinical trials from 27 journals. While the majority of trials mentioned rigor criteria, essential details were often missing. Sixty percent (confidence interval [CI] 53-68%) of trials reported sample size calculations, but only 32% (CI 25-39%) justified the expected effect size. Few trials indicated the blinding status of all main stakeholders (4%; CI 1-7%). Only 18% (CI 12-24%) included information on randomization type, method, and concealed allocation. Most trials reported participants' sex/gender (95%; CI 92-98%) and information on inclusion and exclusion criteria (78%; CI 72-84%). Only 20% (CI 14-26%) of trials were pre-registered. No trials deposited data in open repositories. Conclusions: These results will aid the sports medicine and orthopedics community in developing tailored interventions to improve reporting. While authors typically mention blinding, randomization and other factors, essential details are often missing. Greater acceptance of open science practices, like pre-registration and open data, is needed. These practices have been widely encouraged, we discuss systemic interventions that may improve clinical trial reporting. Registration: https://doi.org/10.17605/OSF.IO/9648H

scholarly journals ­Questionable Research Practices and Open Science in Quantitative Criminology

2021 ◽  
Author(s):  
Jason Chin ◽  
Justin T. Pickett ◽  
Simine Vazire ◽  
Alex O. Holcombe
Keyword(s):  
Open Science ◽  
Research Practices ◽  
Questionable Research Practices

scholarly journals Teaching Good Research Practices: Protocol of a Research Master Course

2019 ◽  
Vol 19 (1) ◽  
pp. 46-59
Author(s):  
Alexandra Sarafoglou ◽  
Suzanne Hoogeveen ◽  
Dora Matzke ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Open Science ◽  
Recent Book ◽  
Practical Training ◽  
Current Crisis ◽  
Research Practices ◽  
Science Practices ◽  
Replication Studies ◽  
Public Sharing ◽  
Social Media Platforms ◽  
Good Research

The current crisis of confidence in psychological science has spurred on field-wide reforms to enhance transparency, reproducibility, and replicability. To solidify these reforms within the scientific community, student courses on open science practices are essential. Here we describe the content of our Research Master course “Good Research Practices” which we have designed and taught at the University of Amsterdam. Supported by Chambers’ recent book The 7 Deadly Sins of Psychology, the course covered topics such as QRPs, the importance of direct and conceptual replication studies, preregistration, and the public sharing of data, code, and analysis plans. We adopted a pedagogical approach that: (a) reduced teacher-centered lectures to a minimum; (b) emphasized practical training on open science practices; and (c) encouraged students to engage in the ongoing discussions in the open science community on social media platforms.

scholarly journals Registered Reports in Special Education: Introduction to the Special Series

2021 ◽  
pp. 074193252199645
Author(s):  
Bryan G. Cook ◽  
Daniel M. Maggin ◽  
Rachel E. Robertson
Keyword(s):  
Special Education ◽  
Publication Bias ◽  
Innovative Approach ◽  
Additional Time ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Constructive Feedback ◽  
Special Series ◽  
Research Questions

This article introduces a special series of registered reports in Remedial and Special Education. Registered reports are an innovative approach to publishing that aim to increase the credibility of research. Registered reports are provisionally accepted for publication before a study is conducted, based on the importance of the research questions and the rigor of the proposed methods. If provisionally accepted, the journal agrees to publish the study if researchers adhere to accepted plans and report the study appropriately, regardless of study findings. In this article, we describe how registered reports work, review their benefits (e.g., combatting questionable research practices and publication bias, allowing expert reviewers to provide constructive feedback before a study is conducted) and limitations (e.g., requires additional time and effort, cannot be applied to all studies), review the application of registered reports in education and special education, and make recommendations for implementing registered reports in special education.

scholarly journals Privacy versus Open Science

2019 ◽  
Author(s):  
Simon Dennis ◽  
Paul Michael Garrett ◽  
Hyungwook Yim ◽  
Jihun Hamm ◽  
Adam F Osth ◽  
...  
Keyword(s):  
Open Science ◽  
Privacy Preserving ◽  
Research Practices ◽  
Psychological Science ◽  
Replication Crisis ◽  
Good Research

Pervasive internet and sensor technologies promise to revolutionize psychological science. However, the data collected using these technologies is often very personal - indeed the value of the data is often directly related to how personal it is. At the same time, driven by the replication crisis, there is a sustained push to publish data to open repositories. These movements are in fundamental conflict. In this paper, we propose a way to navigate this issue. We argue that there are significant advantages to be gained by ceding the ownership of data to the participants who generate it. Then we provide desiderata for a privacy-preserving platform. In particular, we suggest that researchers should use an interface to perform experiments and run analyses rather than observing the stimuli themselves. We argue that this method not only improves privacy but will also encourage greater compliance with good research practices than is possible with open repositories.

scholarly journals Registered Reports in Special Education: Introduction to the Special Series

2021 ◽  
Author(s):  
Bryan G. Cook ◽  
Daniel Maggin ◽  
Rachel Robertson
Keyword(s):  
Special Education ◽  
Publication Bias ◽  
Innovative Approach ◽  
Additional Time ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Constructive Feedback ◽  
Special Series ◽  
Research Questions

This paper introduces a special series of registered reports in Remedial and Special Education. Registered reports are an innovative approach to publishing that aim to increase the credibility of research. Registered reports are provisionally accepted for publication before a study is conducted, based on the importance of the research questions and the rigor of the proposed methods. If provisionally accepted, the journal agrees to publish the study if researchers adhere to accepted plans and report the study appropriately, regardless of study findings. In this article, we describe how registered reports work, review their benefits (e.g., combatting questionable research practices and publication bias, allowing expert reviewers to provide constructive feedback before a study is conducted) and limitations (e.g., requires additional time and effort, cannot be applied to all studies), review the application of registered reports in education and special education, and make recommendations for implementing registered reports in special education.

Sign in / Sign up

Export Citation Format

Share Document