Sustainable Water and Sanitation in Sierra Leone: Impact evaluation of the ‘Improved WASH Services in WAU and WAR Districts’ project

Between October 2016 and March 2019, the Freetown WASH Consortium, led by Oxfam with Against Hunger, Concern Worldwide and Save the Children as members, carried out the 'Improved WASH Services in Western Area Urban (WAU) and Western Area Rural (WAR) Districts' project. Broadly, the project aimed to improve the availability, accessibility, affordability and sustainability of integrated water, sanitation and hygiene (WASH) services, in alignment with the Government of Sierra Leone's national agenda on Ebola recovery and increased preparedness against possible future outbreaks. This Effectiveness Review evaluates the success of this project to increase the sustainability of water and sanitation systems and services. Using a quasi-experimental, mixed method evaluation design, impact is assessed among individuals, households and communities in intervention and comparison areas. Community-level factors contributing to better individual- and household-level outcomes are explored. Find out more by reading the full report now.

Community Currencies as Crisis Response: Results From a Randomized Control Trial in Kenya

This paper presents the results of what may be the world’s first randomized control trial on community currencies, focusing on Grassroots Economics Community Inclusion Currency (CIC) model run on the xDAI blockchain. Beneficiaries in Nairobi, Kenya were sent the equivalent of $30 in cryptocurrency tokens, enabling a level of impact evaluation usually unfeasible for most cash and mobile-money based transfer programs. Results show that CIC transfers of $30 are associated with $93.51 increase in beneficiaries wallet balance, a $23.17 increase in monthly CIC income, a $16.30 increase in monthly CIC spending, a $6.31 increase in average trade size and a $28.43 increase in expenditure on food and water. However, the difference in treatment effects for males versus females suggests gender imbalances persist. This study serves as an important prototype for novel cash transfer models and presents some of the first quantitative evidence in the area of “crypto for good.”

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength

IntroductionAssessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Evaluation in Impact Investing

The emergence of impact investing over the past decade has been accompanied by an increased interest of practitioners and scholars in the impact evaluation topic, one of the twofold pillars of the such an innovative financial approach. To contribute to the international debate, this study adopts a qualitative approach by obtaining results from a systematic literature review of extant research. This is useful to 1) identify the current existing impact evaluation approaches adopted in the field and 2) derive an empirical analysis in the impact investing sector with a focus on impact measurement in social impact bonds. The study opens interesting insights into recognizing the potential for the whole impact investing field, deriving both from theory and evidence of impact evaluation practices.

Co-creating rehabilitation: Findings from a pilot and implications for wider public service reform

As part of a large pan-European project on co-creating public services we supported the design of a programme in England that attempted to operationalise research on desistance, through a model of co-created, strengths-based working. We then evaluated its implementation and impact. The programme was implemented in a Community Rehabilitation Company. It was delivered in the context of rapid organisational change, often in response to rapidly changing external events and a turbulent policy environment. These factors impeded implementation. An impact evaluation did not identify a statistically significant difference in re-offending rates between the intervention group and a comparator group. However, in-depth qualitative evaluation identified positive examples of co-production and co-creation, with individual case managers and service users supportive and noting positive change. Taken as a whole our findings suggest that a co-created, strengths-based model of probation case management is promising but needs to be accompanied by wider systems change if it is to be embedded successfully.

Empirical Analysis of Evidence-Based Policymaking in R&D Programmes

This study explores evidence-based policymaking (EBP) in public research and development (R&D) programmes. Governments develop and implement R&D programmes to promote innovation, the key driver of sustainable economic growth. For effective management, public policies should be planned, executed, and terminated based on their impact evaluation and feedback. Although many studies have investigated the impact of R&D support, few have analysed whether it evolves on the basis of evidence. Using a dataset of Korean programmes, this study adopted the propensity score matching with a difference-in-differences method. We distinguished four determinants of performance evidence: R&D intensity, assets, sales, and profits growth. The following are the main results: (1) while R&D intensity, sales, and profits growth reveal the efficiency of the EBP mechanism, the performance of assets do not differ across the abolished and continued programmes; (2) the EBP process classified by subsidy amount reveals no statistically significant differences in terms of R&D intensity and profits. This suggests that policymakers need to consider the amount of subsidy granted under the budget limitation. This study contributes to the empirical research on EBP using heterogeneous evidence indicators and describes how policymakers exploit policy evaluation to implement and monitor policies for sustainable development.