Achieving high-quality impact evaluation design through mixed methods: the case of infrastructure

2011 ◽  
Vol 3 (1) ◽  
pp. 131-144 ◽  
Author(s):  
Howard White
Keyword(s):  
Mixed Methods ◽  
Impact Evaluation ◽  
Evaluation Design ◽  
High Quality

scholarly journals REducing unwarranted variation in the Delivery of high qUality hip fraCture services in England and Wales (REDUCE): protocol for a mixed-methods study

BMJ Open ◽  
2021 ◽  
Vol 11 (5) ◽  
pp. e049763
Author(s):  
Rita Patel ◽  
Sarah Drew ◽  
Antony Johansen ◽  
Tim Chesser ◽  
Muhammad K Javaid ◽  
...  
Keyword(s):  
Mixed Methods ◽  
Hip Fracture ◽  
Patient Outcomes ◽  
Ethics Committee ◽  
High Quality ◽  
England And Wales ◽  
Fracture Care ◽  
Unwarranted Variation ◽  
Organisational Factors ◽  
Organisational Level

IntroductionSubstantial variation in the delivery of hip fracture care, and patient outcomes persists between hospitals, despite established UK national standards and guidelines. Patients’ outcomes are partly explained by patient-level risk factors, but it is hypothesised that organisational-level factors account for the persistence of unwarranted variation in outcomes. The mixed-methods REducing unwarranted variation in the Delivery of high qUality hip fraCture services in England and Wales (REDUCE) study, aims to determine key organisational factors to target to improve patient care.Methods and analysisQuantitative analysis will assess the outcomes of patients treated at 172 hospitals in England and Wales (2016–2019) using National Hip Fracture Database data combined with English Hospital Episodes Statistics; Patient Episode Database for Wales; Civil Registration (deaths) and multiple organisational-level audits to characterise each service provider. Statistical analyses will identify which organisational factors explain variation in patient outcomes, and typify care pathways with high-quality consistent patient outcomes. Documentary analysis of 20 anonymised British Orthopaedic Association hospital-initiated peer-review reports, and qualitative interviews with staff from four diverse UK hospitals providing hip fracture care, will identify barriers and facilitators to care delivery. The COVID-19 pandemic has posed a major challenge to the resilience of services and interviews will explore strategies used to adapt and innovate. This system-wide understanding will inform the development, in partnership with key national stakeholders, of an ‘Implementation Toolkit’ to inform and improve commissioning and delivery of hip fracture services.Ethics and disseminationThis study was approved: quantitative study by London, City and East Research Ethics Committee (20/LO/0101); and qualitative study by Faculty of Health Sciences University of Bristol Research Ethics Committee (Ref: 108284), National Health Service (NHS) Health Research Authority (20/HRA/71) and each NHS Trust provided Research and Development approval. Findings will be disseminated through scientific conferences, peer-reviewed journals and online workshops.

An impact evaluation design for the Millennium Villages Project in Northern Ghana

2013 ◽  
Vol 5 (2) ◽  
pp. 137-157 ◽  
Author(s):  
Edoardo Masset ◽  
Arnab Acharya ◽  
Chris Barnett ◽  
Tony Dogbe
Keyword(s):  
Impact Evaluation ◽  
Northern Ghana ◽  
Evaluation Design ◽  
Millennium Villages Project

scholarly journals Quality of Inpatient Tuberculosis Health Care in High-Burden Resource-Limited Settings: Protocol for a Comprehensive Mixed Methods Assessment Study

10.2196/13903 ◽  
2020 ◽  
Vol 9 (1) ◽  
pp. e13903
Author(s):  
Nune Truzyan ◽  
Zaruhi Grigoryan ◽  
Lusine Musheghyan ◽  
Byron Crape ◽  
Varduhi Petrosyan
Keyword(s):  
Health Care ◽  
Mixed Methods ◽  
International Standards ◽  
World Health ◽  
High Quality ◽  
High Burden ◽  
Resource Limited ◽  
Evaluation Instruments ◽  
Depth Interviews

Background The quality of care for tuberculosis (TB) is deficient in high-burden countries and urgently needs improvement. However, comprehensively identifying the required improvements is challenging. Providing high-quality TB care is an important step toward improving patients’ quality of life and decreasing TB morbidity and mortality. Effective tools for assessing the quality of TB services using international standards and guidelines can identify existing gaps in services and inform improvements to ensure high-quality inpatient TB services. Objective This study aimed to develop evaluation instruments for defining the quality of provision of TB services. Methods To assess quality of services in the largest TB hospital in Armenia, we developed instruments based on the Joint Commission International Accreditation Standards for Hospitals, International Standards for TB Care, TB Laboratories Bio-Safety Standards, and the World Health Organization framework for conducting TB program reviews. A mixed methods approach was utilized, triangulating quantitative (checklists) and qualitative (in-depth interviews) results. A scoring system and strengths, weaknesses, opportunities, and treats analysis was applied to detail results for each of the 122 standards assessed. A scaling approach was used to present overall performances of inpatient services for eight patient-centered functions and five organization management functions. Results Overall, 40 in-depth interviews and 91 checklists (21 observations, 16 policy papers, 20 staff qualification documents, and 34 medical records) were developed, utilized, and analyzed to explore practices of health care professionals, assess inpatient treatment experience of patients and their family members, evaluate facility environmental conditions, and define the degree of compliance to standards. Conclusions The effective comprehensive evaluation instruments and methods developed in this study for quality of inpatient TB services support the implementation of similar effective assessments in other countries. It may also become a platform to develop similar approaches for assessing ambulatory TB services in resource-limited countries. International Registered Report Identifier (IRRID) DERR1-10.2196/13903

scholarly journals Models of maternal and child healthcare for African refugees: protocol for an exploratory, mixed-methods study

BMJ Open ◽  
2020 ◽  
Vol 10 (6) ◽  
pp. e038162
Author(s):  
Carolyne Njue ◽  
Andrew Hayen ◽  
Angela J Dawson
Keyword(s):  
Mixed Methods ◽  
Health Service ◽  
Service Providers ◽  
Readiness To Change ◽  
Healthcare Services ◽  
Models Of Care ◽  
Nominal Group ◽  
High Quality ◽  
African Refugees ◽  
Service Managers

IntroductionThere is a paucity of research examining models of maternal and child health (MCH) care for refugees in high-income countries. Attention has focused on tailoring existing healthcare services to meet the needs of this population and ensure accessible high-quality patient-centred care. This protocol reports the methodology of a study designed to identify models of care for African refugees in New South Wales (NSW), Australia, to determine the evidence for these models and the improvements necessary to best meet service needs that can be delivered in line with available resources, organisational readiness and capacity to implement.Methods and analysisThis multiphased, participatory research project will employ an exploratory, mixed-methods design. Preparatory activities involve a situational analysis of current MCH services for African refugees and associated policies and guidelines in NSW. We will consult key health service providers and analyse Australian census and settlement data to identify refugee communities and their relation to healthcare services. Phase 1 will ascertain the MCH care needs of African refugees and appropriate service models using: a Delphi survey with health service managers and providers, a nominal group process with African women refugees and; key informant interviews with senior health service managers. This data will be synthesised to provide insight into appropriate models-of-care that could be implemented. These will be discussed in a stakeholder workshop. Phase 2 will comprise a readiness-to-change survey with a selection of providers to explore the willingness, commitment and efficacy of staff members to adopt such models-of-care.Ethics and disseminationEthical approval was granted by NSW Health. Findings will be disseminated to all stakeholders at a knowledge exchange forum to inform the development of a high-quality MCH service delivery model that can be feasibly implemented specifically for African refugee communities.PROSPERO registration numberCRD42018095564.

scholarly journals Exploiting the emergent nature of mixed methods designs: insights from a mixed methods impact evaluation in Malawi

2019 ◽  
Author(s):  
Manuela De Allegri ◽  
Stephan Brenner ◽  
Christabel Kambala ◽  
Jacob Mazalale ◽  
Adamson S Muula ◽  
...  
Keyword(s):  
Mixed Methods ◽  
Data Collection ◽  
Impact Evaluation ◽  
Qualitative Data ◽  
Research Effort ◽  
Methodological Approach ◽  
Maternal And Newborn Health ◽  
Systems Research ◽  
Data Collection And Analysis ◽  
Analytical Approaches

Abstract The application of mixed methods in Health Policy and Systems Research (HPSR) has expanded remarkably. Nevertheless, a recent review has highlighted how many mixed methods studies do not conceptualize the quantitative and the qualitative component as part of a single research effort, failing to make use of integrated approaches to data collection and analysis. More specifically, current mixed methods studies rarely rely on emergent designs as a specific feature of this methodological approach. In our work, we postulate that explicitly acknowledging the emergent nature of mixed methods research by building on a continuous exchange between quantitative and qualitative strains of data collection and analysis leads to a richer and more informative application in the field of HPSR. We illustrate our point by reflecting on our own experience conducting the mixed methods impact evaluation of a complex health system intervention in Malawi, the Results Based Financing for Maternal and Newborn Health Initiative. We describe how in the light of a contradiction between the initial set of quantitative and qualitative findings, we modified our design multiple times to include additional sources of quantitative and qualitative data and analytical approaches. To find an answer to the initial riddle, we made use of household survey data, routine health facility data, and multiple rounds of interviews with both healthcare workers and service users. We highlight what contextual factors made it possible for us to maintain the high level of methodological flexibility that ultimately allowed us to solve the riddle. This process of constant reiteration between quantitative and qualitative data allowed us to provide policymakers with a more credible and comprehensive picture of what dynamics the intervention had triggered and with what effects, in a way that we would have never been able to do had we kept faithful to our original mixed methods design.

scholarly journals A Protocol for a Comprehensive Monitoring and Evaluation Framework With a Compendium of Tools to Assess Quality of Project ECHO (Extension for Community Healthcare Outcomes) Implementation Using Mixed Methods, Developmental Evaluation Design

2021 ◽  
Vol 9 ◽  
Author(s):  
Smita Ghosh ◽  
Brenna M. Roth ◽  
Irene Massawe ◽  
Emmanuel Mtete ◽  
Jacob Lusekelo ◽  
...  
Keyword(s):  
Mixed Methods ◽  
Appreciative Inquiry ◽  
Monitoring And Evaluation ◽  
Evaluation Framework ◽  
Assessment Tools ◽  
Healthcare Outcomes ◽  
Developmental Evaluation ◽  
Evaluation Design ◽  
Community Healthcare ◽  
Project Echo

Introduction: The United States Centers for Disease Control and Prevention (CDC), through U.S. President's Emergency Plan for AIDS Relief (PEPFAR), supports a third of all people receiving HIV care globally. CDC works with local partners to improve methods to find, treat, and prevent HIV and tuberculosis. However, a shortage of trained medical professionals has impeded efforts to control the HIV epidemic in Sub-Saharan Africa and Asia. The Project Extension for Community Healthcare Outcomes (ECHOTM) model expands capacity to manage complex diseases, share knowledge, disseminate best practices, and build communities of practice. This manuscript describes a practical protocol for an evaluation framework and toolkit to assess ECHO implementation.Methods and Analysis: This mixed methods, developmental evaluation design uses an appreciative inquiry approach, and includes a survey, focus group discussion, semi-structured key informant interviews, and readiness assessments. In addition, ECHO session content will be objectively reviewed for accuracy, content validity, delivery, appropriateness, and consistency with current guidelines. Finally, we offer a mechanism to triangulate data sources to assess acceptability and feasibility of the evaluation framework and compendium of monitoring and evaluation tools.Expected impact of the study on public health: This protocol offers a unique approach to engage diverse group of stakeholders using an appreciative inquiry process to co-create a comprehensive evaluation framework and a compendium of assessment tools. This evaluation framework utilizes mixed methods (quantitative and qualitative data collection tools), was pilot tested in Tanzania, and has the potential for contextualized use in other countries who plan to evaluate their Project ECHO implementation.

scholarly journals Sustainable Water and Sanitation in Zambia: Impact evaluation of the 'Urban WASH' project

2021 ◽  
Author(s):  
Jaynie Vonk
Keyword(s):  
Impact Evaluation ◽  
Sustainable Management ◽  
Service Providers ◽  
Evaluation Design ◽  
Water And Sanitation ◽  
Household Level ◽  
Key Informant ◽  
Quasi Experimental ◽  
Sanitation Systems ◽  
Conducive Environment

The ‘Urban WASH' project was implemented in George and Chawama compounds in Lusaka between July 2013 and June 2017 by Oxfam and Village Water Zambia. The project aimed to improve provision and sustainable management of WASH services by engaging citizens to hold duty bearers and service providers to account. Oxfam collaborated with local institutions on an array of activities, engaging stakeholders to create a conducive environment for service provision and improving capacities and practices. This Effectiveness Review evaluates the success of this project to increase the sustainability of water and sanitation systems and services. Using a quasi-experimental evaluation design, we assessed impact among households in the intervention communities and in a comparison community. We combined the household-level quantitative assessment with analysis of community-level qualitative Key Informant Interviews, carried out with relevant institutional representatives. Find out more by reading the full report now.

scholarly journals Design and impact evaluation of a digital reproductive health program in Rwanda using a cluster randomized design: study protocol

2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Cara Nolan ◽  
Laura Packel ◽  
Rebecca Hope ◽  
Jordan Levine ◽  
Laura Baringer ◽  
...  
Keyword(s):  
Reproductive Health ◽  
Hiv Testing ◽  
Impact Evaluation ◽  
Second Phase ◽  
High Quality ◽  
Modern Contraception ◽  
Link Type ◽  
School Based ◽  
Digital Intervention ◽  
Cluster Randomized

Abstract Background Rwandan adolescents have limited access to high-quality family planning and reproductive health (FP/RH) information and care to prevent unplanned pregnancy and HIV/STIs. In addition to the immediate implications for health and well-being, teenage pregnancy is a significant cause of school drop-out, limiting girls’ future potential and employment opportunities. This study introduces a direct-to-consumer digital education program that uses storytelling to deliver age-appropriate FP/RH information and economic empowerment training to adolescents. It also facilitates access to high-quality, youth-friendly FP/RH care and products. We evaluate two different school-based models of its implementation to understand how to optimize the uptake of contraception and HIV testing among adolescents. Methods The study consists of two distinct phases. The first formative intervention design phase, conducted from 2016 to 2019, used a human-centered design methodology to develop the intervention alongside over 600 Rwandan adolescents, their parents, teachers, and healthcare providers. Through this methodology, we sought to maximize the fit between evidence-based practices (uptake of modern contraception and HIV testing) and the implementation context of adolescents in Rwanda. The second phase is an impact evaluation, in which we will use a Hybrid Trial Type 2 Effectiveness-Implementation study design to determine the overall effectiveness of this digital intervention as well as the relative effectiveness of the two different school-based implementation models. This takes the form of a 3-arm cluster-randomized non-inferiority trial, with a sample of 6000 youth aged 12–19 in 60 schools across 8 districts in Rwanda. Primary outcome measures include use of modern contraception, delayed initiation of childbearing, and uptake of HIV testing. Discussion This study will yield insights into not only whether this digital intervention is successful in achieving the intended sexual and reproductive health outcomes, but also which mechanisms are likely to drive this effectiveness. The methodologies used are broadly applicable to the design, implementation, and evaluation of other behavior-based health programs in low and middle-income countries. Trial registration ClinicalTrials.gov Identifier: NCT04198272. Prospectively registered 13 December 2019.

Sign in / Sign up

Export Citation Format

Share Document