Conduct a Scout Survey (Step 6)

We strongly recommend that a ‘scout survey’ of the disaster site be implemented prior to the initiation of the principal study, in order to obtain the types of detailed information required to prepare a research plan and a plan for working with a full research team during the data collection period. The scout survey step requires that one or two researchers go to the disaster site within two or three weeks following the disaster to prepare for a larger team visit, which would initiate work within two to three months post disaster. ‘Scout team’ visits are an essential mechanism for developing and/or revising the data collection instruments, for securing collaboration of local officials who will facilitate the study, for identifying key informants and members of the stakeholder group, and for obtaining background information needed for the sample design. We believe it is essential that the initial data collection be completed as soon as possible after the disaster event ends, in order to minimize memory loss. It is also of great importance that the primary data collection phase be conducted efficiently, within a period of approximately seven to ten days, although additional data may be added subsequent to the primary data collection period as the need for it becomes apparent. It is also likely that a data gathering effort, such as a survey involving large numbers of individuals, may continue after the main data collection team members have returned to their home institutions. The amount of thorough and detailed planning required to achieve the 7–10 day goal virtually mandates a pre-visit by a scout team.

Construct the Logic Model and Research Design (Step 3)

A ‘mixed-methods’ research design, based upon the categories contained in a disaster response logic model, is suggested as the best approach to capture the complexities of the medical and public health disaster response experience. A mixed-methods design allows the evaluation team to collect and combine data from direct observation, medical records, interviews with victims, health professionals, family and friends of victims, public safety officials, other government and non-governmental officials and from public documents. Validation in a mixed-method design is based upon the concept of triangulation. The term triangulation is used in behavioral research to describe the process of obtaining data from three or more different sources and then comparing the findings to assess consistency across sources. In this design, both qualitative and quantitative data are collected and then merged during the analysis phase. Each data set is used to validate and enhance the other in order to improve the validity of the conclusions reached and the recommendations that follow. However, not all data need be combined. The mixed-method design allows for the analysis of certain types of data separately and then applied to the appropriate research question because there may be no appropriate comparative data.

Ethical Considerations in Disaster Evaluation Studies

It is only within the recent past that bioethicists and legal scholars have begun to address the highly complex domain of ethical and legal issues related to the conduct of research in the context of disaster conditions. Since the year 2000, several national and international committees and organizations have developed guidelines for the ethical conduct of research under disaster conditions. These guidelines focus upon three unique characteristics of the disaster situation that researchers must consider: (1) the unique vulnerability of disaster victims; (2) the complications related to obtaining ‘ethics review’ during and immediately after a disaster impact; and (3) threats to research team members, victims and rescue workers during the data collection period. Suggestions for managing these threats and their associated risks are offered in this chapter. The framework and suggestions are drawn from reviews prepared by the World Health Organization and by a team of bioethics and legal scholars who have carefully reviewed and summarized the relevant national and international guideline statements.

Epilogue

This is the epilogue for the book Disaster Evaluation Research.

Analyze Data and Prepare a Final Report (Step 8)

In this chapter we present an overall approach to data analysis and the preparation of a final report. We suggest that the final report should initially be organized to address each major research question. However, as the research plan is implemented, additional questions may emerge or, even more likely, the original questions could have become more nuanced. It is necessary to involve some or all members of the stakeholder groups in the preparation of findings and recommendations. Initially, a simple straightforward presentation of findings, promptly prepared, will best serve the cause of improving medical and public health response efforts. In this chapter we describe a process which will lead to the completion of the type of report that will be useful to stakeholders and decision-makers. It is essential that the report be generated in a timely manner in order to enable decision-makers to draw upon recent public memory to gain support for change where required. Some general comments about data analyses are offered; however, it is not our intent in this book to provide a full discussion about data analysis for mixed-methods studies. It is therefore essential that a methodologist/data analyst be included within the evaluation team.

Construct a Sampling Plan (Step 5)

In disaster studies it is necessary to obtain information from several groups of those involved and affected by the disaster and from various types of medical and administrative documents. We have suggested in Chapter 8 that information be obtained from, at a minimum, (1) survivors/victims/families; (2) professional responders and coordinators (both public safety and EMS/medical); (3) officials of governmental and non-governmental organizations; (4) medical records; and (5) administrative documents. This typically involves accessing a large number of individuals and reports. It is therefore almost always necessary to select a sample from each group and source. When possible some form of random (probability) sampling should be used; a different type of sampling called ‘purposive’ may be employed for key informants. ‘Convenience samples’ are not generally used in scientific evaluation studies due to their great potential for introducing bias into the data. Preparing and implementing a scientific sample design will prove to be one of the most challenging aspects of disaster evaluation studies. It will usually be necessary to consult with an individual who has statistical expertise when preparing the sampling plan; therefore, in this chapter we present some basic concepts in sampling and then conclude with four descriptions of sample designs used in past evaluation studies.

Prepare Mixed-method Data Collection Instruments (Step 4)

This chapter builds upon the response system components described in Chapter 4 and the logic model presented in Chapter 7. The stakeholder group that is assembled to guide the evaluation should determine the evaluation questions that are of greatest interest to them. Topical interview guides are typically prepared to guide interviews with professional responders, victims (or their surrogates), public health, safety, and governmental administrators, and non-governmental organization participants. In addition, an autopsy/hospital record abstract form and an ‘observation guide’ are often included in the set of data collection forms. Data collection forms used in past evaluations can be found in Appendix C and may be useful as examples.

Formulate Evaluation Questions (Step 2)

In this chapter we define the five basic categories of evaluation, namely structure (resources), process (activities), outcomes, adequacy, and costs associated with the response(s). Structure refers to the equipment and personnel and the way in which these resources were organized for use in the medical response. Process refers to the activities carried out during the disaster response. Outcome assessment concerns the results of the care provided on the patients served, usually measured over time. Adequacy describes the extent to which the search-and-rescue, pre-hospital and hospital, and public health responses were able to meet the needs of the community during the disaster response. In general, these categories are consistent with the design of a typical logic model. Following the discussion of ‘evaluation categories’ we suggest questions that the evaluation team might consider for inclusion in the evaluation study. For each category we suggest questions which could be addressed in any disaster evaluation study which focuses on the medical and public health response. The stakeholder group should be fully involved in the selection of questions to be addressed by the evaluation team.

Components of a Disaster Response System

The intent of this chapter is to describe the general framework of emergency public health and emergency medical services operations in disaster events in order to inform the planning and conduct of the evaluation study. The objective of disaster medical and public health response operations is to deliver immediate health care relief to an affected population until ‘pre-event’ health services can be restored. The functions of the health sector described in this chapter are termed basic societal/support functions or emergency support functions. Each function carries with it a series of activities or interventions; therefore, it seems reasonable to identify and study these activities from the perspective of the user rather than from an event-specific viewpoint. It is important to note that the outcomes or level of performance of these activities be assessed both quantitatively and qualitatively within the context of the larger evaluation study. More importantly, the data collected through systematic evaluation in these cases should be uniformly reported so as to make them generalizable to similar hazard events, thereby facilitating the creation of an ‘all-hazards’ repository of information on disasters. In this chapter we describe a generalized disaster response framework, including a ‘concept of operations’, for the public health and medical components of a disaster response system.

Introduction to a Model for Disaster Evaluation

In this chapter we move to a discussion of the determination of objectives for a disaster evaluation study and the rationale behind the design and use of a logic model. Logic models are now widely used throughout the domain of public health as a tool for both designing and evaluating health programs (typically referred to as ‘interventions’). This chapter is offered as a general introduction to the detailed discussion of eight steps for conducting a disaster evaluation as presented in Part 2 of this Field Manual. The selection of two key components of our general evaluation framework, ‘evaluation objectives’ and ‘program theory/logic model’ for further discussion, is to highlight the centrality and importance of these ideas and concepts for scientific evaluations.