scholarly journals Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

2021 ◽  
Author(s):  
Evan Mayo-Wilson ◽  
Sean Grant ◽  
Lauren H. Supplee
Keyword(s):  
Conflicts Of Interest ◽  
Internal Validity ◽  
Open Science ◽  
Evidence Based ◽  
Causal Inferences ◽  
Science Practices ◽  
Social Interventions ◽  
Registration Data ◽  
Standards Of Evidence ◽  
Intervention Evaluations

AbstractClearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

scholarly journals Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

2020 ◽  
Author(s):  
Evan Mayo-Wilson ◽  
Sean Grant ◽  
Lauren Supplee
Keyword(s):  
Conflicts Of Interest ◽  
Open Science ◽  
Research Quality ◽  
Evidence Based ◽  
Science Practices ◽  
Social Interventions ◽  
Registration Data ◽  
Standards Of Evidence ◽  
Study Designs ◽  
Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based”, clearinghouses evaluate empirical research using published standards of evidence that focus on study design features. Study designs that support causal inferences are necessary but insufficient for intervention evaluations to produce true results. The use of open science practices can improve the probability that evaluations produce true results and increase trust in research. In this study, we examined the degree to which the policies, procedures, and practices of 10 federal evidence clearinghouses consider the transparency, openness, and reproducibility of intervention evaluations. We found that seven clearinghouses consider at least one open science practice: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, materials sharing, and citation standards. Clearinghouse processes and standards could be updated to promote research transparency and reproducibility by reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could improve research quality, increase trustworthiness of evidence used for policy making, and support the evidence ecosystem to adopt open science practices.

The Role of Open Science Practices in Scaling Evidence-Based Prevention Programs

2021 ◽  
Author(s):  
Lauren H. Supplee ◽  
Robert T. Ammerman ◽  
Anne K. Duggan ◽  
John A. List ◽  
Dana Suskind
Keyword(s):  
Prevention Programs ◽  
Open Science ◽  
Evidence Based ◽  
Science Practices

scholarly journals Investigating cognitive and therapeutic effects of transcranial electric stimulation (TES): a short guide for reproducible and transparent research

2020 ◽  
Author(s):  
Samuel James Westwood
Keyword(s):  
Random Noise ◽  
Evidence Base ◽  
Open Science ◽  
Transcranial Electrical Stimulation ◽  
Therapeutic Effects ◽  
Direct Stimulation ◽  
Causal Inferences ◽  
Science Practices ◽  
Transcranial Random Noise Stimulation ◽  
Transcranial Alternating Current Stimulation

Non-invasive brain stimulation (NIBS) is a useful tool for assisting causal inferences in cortical structure–function relationships and for developing treatment alternatives in neuropsychiatric disorders. Transcranial electrical stimulation (TES) techniques, namely transcranial direct stimulation (tDCS), transcranial alternating current stimulation (tACS), and transcranial random noise stimulation (tRNS), have grown in popularity in recent years because they are well tolerated, safe, and less costly compared to other forms of NIBS such as transcranial magnetic stimulation (TMS). The rigour and reproducibility of TES studies and their findings is therefore of key importance. However, in recent years, uncertainty has grown regarding the effectiveness of TES and the quality of its evidence base. Current recommended steps to improve the rigour and reproducibility of TES research have mainly focused on reducing variation and ambiguity in reported findings. To effectively deal with reported issues, Open Science practices provide a possible answer. The goal of this article is to give an overview of Open Science practices relevant to TES research (i.e., open materials/data, preregistration, registered reports, collaboration) and guidance in how to overcome challenges one may face in their implementation (e.g., data anonymity, costs of sharing materials/data, inappropriate incentives). The ultimate goal of this article is to instigate more engagement from TES researchers in how Open Science practices can be incorporated into standard research practice.

scholarly journals Opening Doors to Open Science and Scholarship for School Psychology Research, Training, and Practice

2020 ◽  
Author(s):  
Anthony J. Roberson ◽  
Ryan L. Farmer ◽  
Steven Shaw ◽  
Shelley Upton ◽  
Imad Zaheer
Keyword(s):  
School Psychology ◽  
School Psychologists ◽  
Research Training ◽  
Scientific Evidence ◽  
Open Science ◽  
Evidence Based ◽  
Evidence Based Practices ◽  
Questionable Research Practices ◽  
Science Practices ◽  
Research Findings

Trustworthy scientific evidence is essential if school psychologists are to use evidence-based practices to solve the big problems students, teachers, and schools face. Open science practices promote transparency, accessibility, and robustness of research findings, which increases the trustworthiness of scientific claims. Simply, when researchers, trainers, and practitioners can ‘look under the hood’ of a study, (a) the researchers who conducted the study are likely to be more cautious, (b) reviewers are better able to engage the self-correcting mechanisms of science, and (c) readers have more reason to trust the research findings. We discuss questionable research practices that reduce the trustworthiness of evidence; specific open science practices; applications specific to researchers, trainers, and practitioners in school psychology; and next steps in moving the field toward openness and transparency.

scholarly journals Transparent, Open, and Reproducible Prevention Science

2020 ◽  
Author(s):  
Sean Grant ◽  
Kathleen Wendt ◽  
Bonnie J. Leadbeater ◽  
Lauren H. Supplee ◽  
Evan Mayo-Wilson ◽  
...  
Keyword(s):  
Scientific Practice ◽  
Scientific Discovery ◽  
Program Planning ◽  
Well Being ◽  
Open Science ◽  
Prevention Science ◽  
Evidence Based ◽  
Science Practices ◽  
Research Activities ◽  
Science Movement

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in transparency, openness, and reproducibility. Open science provides opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. Open science also addresses key challenges to the credibility of prevention science, such as irreproducibility of results, selective non-reporting (publication bias, outcome reporting bias), and other detrimental research practices. The overarching goal of this paper is to provide an overview of open science practices for prevention science researchers, and to identify key stakeholders and resources to support implementation of these practices. We consider various aspects of applying open science practices in prevention science, such as identifying evidence-based interventions. In addition, we call for the adoption of prevention science practices in the open science movement, such as the use of program planning principles to develop, implement, and evaluate open science efforts. We also identify some challenges that need to be considered in the transition to a transparent, open, and reproducible prevention science. Throughout, we identify activities that will strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. We conclude with the notion that prevention scientists are well-positioned to engage with the open science movement, especially given their expertise in examining and addressing complex social and behavioral issues. By embracing transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote well-being.

Evidence-Based Social Sciences: A New Emerging Field

2018 ◽  
Vol 5 (2) ◽  
pp. 207-211
Author(s):  
Nazila Zarghi ◽  
Soheil Dastmalchian Khorasani
Keyword(s):  
Social Sciences ◽  
Critical Appraisal ◽  
State Of The Art ◽  
Internal Validity ◽  
Evidence Based ◽  
Multiple Sources ◽  
Social Settings ◽  
Research Findings ◽  
Academic Area ◽  
Different Levels

Abstract Evidence based social sciences, is one of the state-of- the-art area in this field. It is making decisions on the basis of conscientious, explicit and judicious use of the best available evidence from multiple sources. It also could be conducive to evidence based social work, i.e a kind of evidence based practice in some extent. In this new emerging field, the research findings help social workers in different levels of social sciences such as policy making, management, academic area, education, and social settings, etc.When using research in real setting, it is necessary to do critical appraisal, not only for trustingon internal validity or rigor methodology of the paper, but also for knowing in what extent research findings could be applied in real setting. Undoubtedly, the latter it is a kind of subjective judgment. As social sciences findings are highly context bound, it is necessary to pay more attention to this area. The present paper tries to introduce firstly evidence based social sciences and its importance and then propose criteria for critical appraisal of research findings for application in society.

Journal Editor and Associate Editor Perspectives on Research Reproducibility and Open Science

2021 ◽  
pp. 074193252110172
Author(s):  
Daniel M. Maggin
Keyword(s):  
Social Sciences ◽  
Special Education ◽  
Journal Editor ◽  
Associate Editor ◽  
Open Science ◽  
Science Practices ◽  
Journal Editors ◽  
Education School ◽  
Special Education School ◽  
Insight Into

Interest in transparent and open science is increasing in special education, school psychology, and related disciplines. Proponents for open science reforms provide evidence that researchers in special education, and the broader social sciences, engage in practices that mitigates its credibility and reduces the validity of information disseminated to practitioners and policymakers. In light of these issues, this article reports on a survey of journal editors-in-chief and associate editors to gain insight into concerns regarding research reproducibility, and the familiarity and viability of open science for improving research credibility. Results indicate that respondents were concerned about research reproducibility, were moderately familiar with open science practices, and viewed many as effective for improving research credibility. Finally, respondents supported the use of journals to encourage open science practices though there was little support for requiring their use. Findings are discussed in relation to open science and implications for research and practice.

scholarly journals Recommendations for an Open Science approach to welding process research data

2021 ◽  
Author(s):  
Cagtay Fabry ◽  
Andreas Pittner ◽  
Volker Hirthammer ◽  
Michael Rethmeier
Keyword(s):  
Domain Knowledge ◽  
Knowledge Exchange ◽  
Welding Process ◽  
Open Science ◽  
Research Data ◽  
Fusion Welding ◽  
File Format ◽  
Data Format ◽  
Science Practices ◽  
Science Community

AbstractThe increasing adoption of Open Science principles has been a prevalent topic in the welding science community over the last years. Providing access to welding knowledge in the form of complex and complete datasets in addition to peer-reviewed publications can be identified as an important step to promote knowledge exchange and cooperation. There exist previous efforts on building data models specifically for fusion welding applications; however, a common agreed upon implementation that is used by the community is still lacking. One proven approach in other domains has been the use of an openly accessible and agreed upon file and data format used for archiving and sharing domain knowledge in the form of experimental data. Going into a similar direction, the welding community faces particular practical, technical, and also ideological challenges that are discussed in this paper. Collaboratively building upon previous work with modern tools and platforms, the authors motivate, propose, and outline the use of a common file format specifically tailored to the needs of the welding research community as a complement to other already established Open Science practices. Successfully establishing a culture of openly accessible research data has the potential to significantly stimulate progress in welding research.

scholarly journals Learning Open Science by doing Open Science. A reflection of a qualitative research project-based seminar

2020 ◽  
Vol 36 (3) ◽  
pp. 263-279
Author(s):  
Isabel Steinhardt
Keyword(s):  
Qualitative Research ◽  
Open Source ◽  
Knowledge Society ◽  
Open Data ◽  
Open Science ◽  
Research Project ◽  
Science Practices ◽  
Bachelor's Degrees ◽  
Participatory Technologies ◽  
Digital Knowledge

Openness in science and education is increasing in importance within the digital knowledge society. So far, less attention has been paid to teaching Open Science in bachelor’s degrees or in qualitative methods. Therefore, the aim of this article is to use a seminar example to explore what Open Science practices can be taught in qualitative research and how digital tools can be involved. The seminar focused on the following practices: Open data practices, the practice of using the free and open source tool “Collaborative online Interpretation, the practice of participating, cooperating, collaborating and contributing through participatory technologies and in social (based) networks. To learn Open Science practices, the students were involved in a qualitative research project about “Use of digital technologies for the study and habitus of students”. The study shows the practices of Open Data are easy to teach, whereas the use of free and open source tools and participatory technologies for collaboration, participation, cooperation and contribution is more difficult. In addition, a cultural shift would have to take place within German universities to promote Open Science practices in general.

Sign in / Sign up

Export Citation Format

Share Document