The Role of Open Science Practices in Scaling Evidence-Based Prevention Programs

2021 ◽  
Author(s):  
Lauren H. Supplee ◽  
Robert T. Ammerman ◽  
Anne K. Duggan ◽  
John A. List ◽  
Dana Suskind
Keyword(s):  
Prevention Programs ◽  
Open Science ◽  
Evidence Based ◽  
Science Practices

Open science reforms: Strengths, challenges, and future directions

2021 ◽  
Author(s):  
Kathryn R. Wentzel
Keyword(s):  
Research Design ◽  
Educational Psychology ◽  
Open Science ◽  
Policy And Practice ◽  
Future Directions ◽  
Science Practices ◽  
Research Findings ◽  
Potential Benefits ◽  
The Impact

In this article, I comment on the potential benefits and limitations of open science reforms for improving the transparency and accountability of research, and enhancing the credibility of research findings within communities of policy and practice. Specifically, I discuss the role of replication and reproducibility of research in promoting better quality studies, the identification of generalizable principles, and relevance for practitioners and policymakers. Second, I suggest that greater attention to theory might contribute to the impact of open science practices, and discuss ways in which theory has implications for sampling, measurement and research design. Ambiguities concerning the aims of preregistration and registered reports also are highlighted. In conclusion, I discuss structural roadblocks to open science reform and reflect on the relevance of these reforms for educational psychology.

scholarly journals Researcher Degrees of Freedom in the Psychology of Religion

2018 ◽  
Author(s):  
Sarah Jane Charles ◽  
James Edward Bartlett ◽  
Kyle J. Messick ◽  
Thomas Joseph Coleman ◽  
Alex Uzdavines
Keyword(s):  
Systematic Review ◽  
Degrees Of Freedom ◽  
Psychology Of Religion ◽  
Open Science ◽  
Science Practices ◽  
Replication Crisis ◽  
The Way

There is a push in psychology toward more transparent practices, stemming partially as a response to the replication crisis. We argue that the psychology of religion should help lead the way toward these new, more transparent practices to ensure a robust and dynamic subfield. One of the major issues that proponents of Open Science practices hope to address is researcher degrees of freedom (RDF). We pre-registered and conducted a systematic review of the 2017 issues from three psychology of religion journals. We aimed to identify the extent to which the psychology of religion has embraced Open Science practices and the role of RDF within the subfield. We found that many of the methodologies that help to increase transparency, such as pre-registration, have yet to be adopted by those in the subfield. In light of these findings, we present recommendations for addressing the issue of transparency in the psychology of religion and outline how to move toward these new Open Science practices.

scholarly journals Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

2020 ◽  
Author(s):  
Evan Mayo-Wilson ◽  
Sean Grant ◽  
Lauren Supplee
Keyword(s):  
Conflicts Of Interest ◽  
Open Science ◽  
Research Quality ◽  
Evidence Based ◽  
Science Practices ◽  
Social Interventions ◽  
Registration Data ◽  
Standards Of Evidence ◽  
Study Designs ◽  
Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based”, clearinghouses evaluate empirical research using published standards of evidence that focus on study design features. Study designs that support causal inferences are necessary but insufficient for intervention evaluations to produce true results. The use of open science practices can improve the probability that evaluations produce true results and increase trust in research. In this study, we examined the degree to which the policies, procedures, and practices of 10 federal evidence clearinghouses consider the transparency, openness, and reproducibility of intervention evaluations. We found that seven clearinghouses consider at least one open science practice: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, materials sharing, and citation standards. Clearinghouse processes and standards could be updated to promote research transparency and reproducibility by reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could improve research quality, increase trustworthiness of evidence used for policy making, and support the evidence ecosystem to adopt open science practices.

scholarly journals Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

2021 ◽  
Author(s):  
Evan Mayo-Wilson ◽  
Sean Grant ◽  
Lauren H. Supplee
Keyword(s):  
Conflicts Of Interest ◽  
Internal Validity ◽  
Open Science ◽  
Evidence Based ◽  
Causal Inferences ◽  
Science Practices ◽  
Social Interventions ◽  
Registration Data ◽  
Standards Of Evidence ◽  
Intervention Evaluations

AbstractClearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

scholarly journals Opening Doors to Open Science and Scholarship for School Psychology Research, Training, and Practice

2020 ◽  
Author(s):  
Anthony J. Roberson ◽  
Ryan L. Farmer ◽  
Steven Shaw ◽  
Shelley Upton ◽  
Imad Zaheer
Keyword(s):  
School Psychology ◽  
School Psychologists ◽  
Research Training ◽  
Scientific Evidence ◽  
Open Science ◽  
Evidence Based ◽  
Evidence Based Practices ◽  
Questionable Research Practices ◽  
Science Practices ◽  
Research Findings

Trustworthy scientific evidence is essential if school psychologists are to use evidence-based practices to solve the big problems students, teachers, and schools face. Open science practices promote transparency, accessibility, and robustness of research findings, which increases the trustworthiness of scientific claims. Simply, when researchers, trainers, and practitioners can ‘look under the hood’ of a study, (a) the researchers who conducted the study are likely to be more cautious, (b) reviewers are better able to engage the self-correcting mechanisms of science, and (c) readers have more reason to trust the research findings. We discuss questionable research practices that reduce the trustworthiness of evidence; specific open science practices; applications specific to researchers, trainers, and practitioners in school psychology; and next steps in moving the field toward openness and transparency.

scholarly journals Transparent, Open, and Reproducible Prevention Science

2020 ◽  
Author(s):  
Sean Grant ◽  
Kathleen Wendt ◽  
Bonnie J. Leadbeater ◽  
Lauren H. Supplee ◽  
Evan Mayo-Wilson ◽  
...  
Keyword(s):  
Scientific Practice ◽  
Scientific Discovery ◽  
Program Planning ◽  
Well Being ◽  
Open Science ◽  
Prevention Science ◽  
Evidence Based ◽  
Science Practices ◽  
Research Activities ◽  
Science Movement

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in transparency, openness, and reproducibility. Open science provides opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. Open science also addresses key challenges to the credibility of prevention science, such as irreproducibility of results, selective non-reporting (publication bias, outcome reporting bias), and other detrimental research practices. The overarching goal of this paper is to provide an overview of open science practices for prevention science researchers, and to identify key stakeholders and resources to support implementation of these practices. We consider various aspects of applying open science practices in prevention science, such as identifying evidence-based interventions. In addition, we call for the adoption of prevention science practices in the open science movement, such as the use of program planning principles to develop, implement, and evaluate open science efforts. We also identify some challenges that need to be considered in the transition to a transparent, open, and reproducible prevention science. Throughout, we identify activities that will strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. We conclude with the notion that prevention scientists are well-positioned to engage with the open science movement, especially given their expertise in examining and addressing complex social and behavioral issues. By embracing transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote well-being.

scholarly journals (Re)Building Trust? Journals’ Open Science Badges Influence Trust in Scientists.

2020 ◽  
Author(s):  
Jürgen Schneider ◽  
Tom Rosman ◽  
Augustin Kelava ◽  
Samuel Merk
Keyword(s):  
Journal Article ◽  
Effective Means ◽  
Open Science ◽  
Epistemic Beliefs ◽  
Science Standards ◽  
Science Practices ◽  
Replication Crisis ◽  
Title Pages ◽  
Moderating Role

As a response to the replication crisis, reforms call for the implementation of open science standards. In this regard, open science badges are a promising method to signal a study’s adherence to open science practices (OSP). In an experimental study, we investigated whether badges on journal article title pages affect non-scientists’ trust in scientists. Furthermore, we analyzed the moderating role of epistemic beliefs in this regard. We randomly assigned 270 non-scientists to two of three conditions: Badges awarded (visible compliance to OSP), badges not awarded (visible non-compliance to OSP) and no badges (compliance not visible, control condition). Results indicate that badges influence trust in scientists as well as the epistemic beliefs of participants. However, epistemic beliefs did not moderate the effect of badges on trust. In sum, our paper provides support to the notion that badges are an effective means to promote epistemic beliefs and trust in scientists.

scholarly journals Improving expert evidence: the role of open science and transparency

2018 ◽  
Author(s):  
Jason Chin ◽  
Bethany Growns ◽  
David Thomas Mellor
Keyword(s):  
Open Science ◽  
Expert Witnesses ◽  
Evidence Based ◽  
Expert Evidence ◽  
Wrongful Convictions ◽  
Disclosure Requirements ◽  
Trial Judges ◽  
Evidence Law ◽  
Science Movement

Both science and expert evidence law are undergoing significant changes. In this article, the authors compare these two movements – the open science movement and the evidence-based evidence movement. The open science movement encompasses the recent discovery of many irreproducible findings in science and the subsequent move towards more transparent methods. The evidence-based evidence movement is the discovery that many forms of expert evidence are unreliable, and that they have contributed to wrongful convictions. The authors identify similarities between these movements, which suggest how courts and legal actors may learn from the open science movement to produce more accurate results. Expert witnesses should comport themselves as rigorous open scientists to produce evidence that is more susceptible to evaluation. Parties should be subjected to more specific and rigorous disclosure requirements because research has shown that even leading scientists find it easy to discount and suppress findings that do not support their hypotheses. And trial judges, as gatekeepers, should not defer to the generally accepted practices that have proven insufficient in the mainstream sciences.

Sign in / Sign up

Export Citation Format

Share Document