Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based”, clearinghouses evaluate empirical research using published standards of evidence that focus on study design features. Study designs that support causal inferences are necessary but insufficient for intervention evaluations to produce true results. The use of open science practices can improve the probability that evaluations produce true results and increase trust in research. In this study, we examined the degree to which the policies, procedures, and practices of 10 federal evidence clearinghouses consider the transparency, openness, and reproducibility of intervention evaluations. We found that seven clearinghouses consider at least one open science practice: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, materials sharing, and citation standards. Clearinghouse processes and standards could be updated to promote research transparency and reproducibility by reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could improve research quality, increase trustworthiness of evidence used for policy making, and support the evidence ecosystem to adopt open science practices.

An example of involving lay people to enhance complex public health intervention evaluations

Background Despite growing interest in understanding complex systems and public health interventions, research methodologies that take account of system-wide action are relatively underdeveloped. Community-based participatory research (CBPR) is steered and conducted by people with lived experience of the issues being researched. This paper explores the value of CBPR in complex public health intervention evaluations. The 'Local People' and 'Local Conversations' programmes use a community empowerment approach in 50 communities across the UK experiencing social disadvantage to increase social connections and collective control, improve health and wellbeing, and reduce inequalities (linked to SDG 3 and 11). Methods Evaluation of the programmes followed a mixed-methods design, including qualitative case studies, longitudinal survey, process appraisal, and CBPR. Residents from 10 communities across the programmes each undertook 2 rounds of CBPR. These projects resulted in written reports, which were analysed thematically alongside other data sources. Results There was some variation in the scope and design of the 20 completed CBPR projects. Whilst projects did not generally extend beyond the scope of the overall evaluation, peer research provided information from residents that were inaccessible to other data collection streams. Gathering community (lay) knowledge improved understanding of local priorities and actions within the programmes. However, the utility of CBPR was less consistent for community-researchers and local communities, often failing to support project development. Some community-researchers felt unprepared for the activity despite support from the academic team. Conclusions Conducted appropriately, CBPR can elicit data that would be less accessible through externally led research. This study highlights the value of CBPR in complex programme evaluations, enabling a deeper understanding of social context in which interventions occur. Key messages CBPR complements more traditional research methodologies in complex public health evaluation designs. CBPR can enable a deeper understanding of social processes necessary for the success of complex public health interventions that might be beyond the scope of other methodologies.

Dog assisted therapy for teenagers with emotional and behavioural issues: A multicentre study

IntroductionFurther research is still needed to demonstrate the benefits of animal-assisted therapy (AAT) for specific participant profiles, such as children with behavioural disorders.ObjectivesWe wanted to find out if AAT could be considered an efficient therapeutic strategy for the treatment of children with behavioural disorders.AimsWe wanted to study the effects of a preestablished AAT program on the behaviour of children with emotional and behavioural issues in 6 different reception centres for children under government guardianship.MethodsForty-five children (12 to 17 years old) with emotional and behavioural issues participated in a 14-session AAT program. Behavioural measures were those routinely scored as part of therapy; an observational report of 3 different problematic behaviours (such as impulsivity, lack of social skills or lack of personal recognition) was made twice a week for each child (with a score of frequency and intensity). A pre- and post-treatment “global behaviour score” was calculated for each child, as an average value of the 3 problematic behaviours measured during the month pre-treatment and the month post-treatment.ResultsThe 45 participants attended, on average, 72.8% of AAT sessions. Independent behaviour scores differed between the pre- and post-intervention evaluations (n = 135 behaviours) (Wilcoxon test; P < 0.0001). Based on the global behaviour score for each child (n = 45), significant change was found between pre- and post-intervention evaluations (Wilcoxon test; P = 0.0011).ConclusionsOur results suggest AAT could be a beneficial intervention for children with behavioural issues in terms of program adherence and behaviour improvement.Disclosure of interestThe authors have not supplied their declaration of competing interest.