Approaching Evaluations of Surveillance System Pilots through an Ownership Perspective

ObjectiveWe used experiences in multiple countries to determine that owner engagement is critical for successful evaluations of surveillance system viability.IntroductionPilot projects help determine utility and feasibility of a system, but even if considered successful, cost could prevent further scale-up. When evaluating a surveillance system pilot, cost and benefits are key factors to examine. In Cote d’Ivoire and Tanzania, Ministry of Health (MoH) and non-governmental partners receive funding under the Global Health Security Agenda to strengthen disease surveillance for earlier detection and improved response to potential infectious disease outbreaks. To this end, Community based surveillance (CBS) projects were implemented in 2016 as a means for early warning of potential events to facilitate a more rapid response.Currently, these CBS projects are being evaluated collaboratively with the primary stakeholder, the host country government (HCG), as lead, and partners such as CDC providing technical assistance. In other instances, partners may conduct an evaluation and share the results and recommendations with the HCG; however, if the HCG is not actively engaged as the primary executor, outcomes may not be endorsed or implemented. Therefore, these evaluations were approached from an owner’s (HCG) perspective. In this way, the governmental agencies develop capabilities to conduct similar activities in other areas, reduce dependencies on outside entities, and promote enactment of resulting recommendations.MethodsOnce the determination was made that an evaluation was necessary to decide the usefulness of the projects for future planning, key stakeholders worked together to design and execute the evaluation. For Cote d’Ivoire, the evaluation team consisted of representatives from the MoH’s National Institute of Public Health, Directorate of Informatics and Health Information, and Directorate for the Coordination of the Expanded Immunization Program, along with delegates from CDC and implementing partners. In Tanzania, evaluation team members came from the MoH, the Ministry of Agriculture, Forestries and Livestock, WHO, CDC and implementing partners. Team members participated in either planning, conducting or analyzing the evaluation, while some contributed to a combination or to all aspects. MoH members led the effort with CDC and other partners providing technical assistance, while implementing partners contributed only to planning and logistics to reduce the potential for bias.For the initial step, representatives came together to fully document the system to be evaluated. This system description details the purpose, relevant stakeholders and current operation of the pilot system. As the evaluation question should remain within the scope of the system’s purpose, it was necessary to definitively understand and confirm the goal and objectives set out for the system. Next, the sites, participants and roles, and data flows were described, noting that verification of the actual processes would occur during the site visit portion of the evaluation. Total cost of ownership was calculated by considering solution costs, implementation costs and ongoing support, and then broken down by district. The CBS pilots implemented early warning notification systems in two districts in Cote d’Ivoire and in five districts in Tanzania using a combination of paper-based and electronic reporting formats.Evaluation teams visited pilot sites and routine surveillance sites for comparison and conducted in-person interviews using questionnaires specific to the individual’s role. Data were either collected in the field on paper forms or electronically on tablets for subsequent upload to a centralized database for later analysis. Data from project and routine reporting databases were comparatively analyzed to calculate timeliness, validity, usefulness, acceptability and value of the early warning system pilots.ResultsAlthough final interpretations of the evaluation results are pending, the evaluations were successfully led by the HCG and jointly conducted with other stakeholder engagement. Leadership by the owners of the systems has already resulted in the recognition that certain aspects of the pilot surveillance systems demonstrate a successful and affordable approach, while others will need to consider more cost-effective strategies. Though further analysis will likely continue to show the utility of CBS strategies, the ownership approach is resulting in an outcome of broad stakeholder input with approval from the host country government.ConclusionsCommunity based surveillance can help to detect events of public health importance and effect earlier introduction into the health system for more timely situational awareness and response. However, it is difficult to determine the costs associated with different strategies of implementation and operation in order to ascertain the value for public health action. Additionally, pilot implementations of these systems are often funded at a level that cannot be replicated nationally and not for a prolonged period of time. While it is believed that CBS can be a cost effective early notification system, continual monitoring and routine evaluation is required. By routinely monitoring cost and quality, sustainability of the system can be continually assessed and system adaptations made accordingly. Key to remember is that evaluation must occur from an owner’s perspective and must engage the people who are going to govern, operate and provide the ongoing resources for system operation. In this way, effectiveness and efficiency can be continually monitored within the parameter of cost so that viability of the system can be ascertained.