A critical realist approach to systems thinking in evaluation

Pawson and Tilley’s acknowledgment of programs embedded in multiple social systems has gained little traction in realist synthesis and evaluation practice. A practice focusing on fairly closed systems—explaining how programs work and do not work—has emerged. This article negotiates the boundaries of knowledge pertinent to have in program design and evaluation from a realist perspective. It highlights critical realism as another possible response to systems thinking in evaluation. Moving one level up a program, it theorizes about social structures, mechanisms, and causes operating in a complex system within which an education-to-work program is nested. Three implications of the approach are highlighted: it foregrounds the relational nature of social, psychological, and programmatic structures and mechanisms; enables policymakers to develop a broader understanding of structures needed to support a program; and enables program architects to ascertain how a planned program might assimilate and adapt to social structures and mechanisms already established in a context.

Program, complexity, and system when evaluating sustainable development

In light of the challenges of the Anthropocene, it is common today for evaluators to propose a systems rather than a program perspective. A program perspective is reproached for its simplicity and linearity, while the systems perspective is praised for its ability to account for complexity and emergence. This article argues that even before the systems perspective appeared on the horizon, the story of program evaluation had been characterized by a confrontation between “simplifiers” and “complexifiers.” It enquires into the lessons that the complexifiers of the evaluation of programs have handed on to evaluators who are facing the current challenges of sustainable development. On the other hand, when analyzing what a systems perspective contributes, the article is alert to the risk that a “holistic” view may ignore lessons complexifiers have to offer. This argument is supported by considering the way in which a systems perspective is understood in the practice of evaluation. Conclusions invite readers to overcome a misleading opposition between the two perspectives and indicate possible cross-fertilization across different units of analysis and approaches.

The unused potential of process tracing as evaluation approach: The case of cluster policy evaluation

This article shows that process tracing developed in social science research can be used in evaluations of complex structural and technology policy programmes to overcome deficits in the methodological instruments used to date. Cluster policies are a well-suited example because they are characterized by complex impact patterns like many other current structural and innovation policy programmes. The origin and characteristics of the methodological approach of process tracing are discussed and weaknesses of impact evaluations of cluster programmes highlighted. Subsequently, we look at the potentials of process tracing in impact evaluations of cluster programmes within the framework of mixed-method designs. Our analysis shows that process tracing can enrich the applied methodological repertoire. It allows the evaluators to test the accuracy of the theoretical assumptions underlying the analysed programme and to identify causal mechanisms.

Using environmental evaluation systems and their contribution to sustainable development

Drawing on previous research into evaluation use and sustainable development, this article investigates to what extent and how a national evaluation system can advance sustainable development. The data comprise interviews with key actors, observations at seminars where evaluations were discussed, and analysis of evaluation reports. The actors were divided into three categories (i.e. central, proximal, and peripheral) to indicate their closeness to the national policy cycle. The results indicate that the evaluations were used mainly by central and proximal actors, primarily because the evaluation knowledge met most of their knowledge needs, and very little used by peripheral actors, such as environmental and business organizations. The evaluation system’s main influence was to consolidate environmental work and provide guidance on the environmental dimension of sustainable development at the national level. The evaluation system also served as a recurring reminder for stakeholders about the national objectives, their achievement, and everyone’s responsibility for their implementation.

Preventing children’s death by heatstroke: How alternative accident theories affect policy design and evaluation

When public policies are designed from scratch, with no or limited possibility of learning from past experience, decision makers rely on their creativity and lateral thinking to counterbalance cognitive biases. One of the crucial issues for both evaluators and policy makers in such situations is not only to elaborate a sound theory of the programme but also to be aware of the existence of competing theories to solve the same problem. Being able to detect such theories constitutes an essential step to avoid ‘Type III errors’, that is, enacting the right solution for the ‘wrong’ problem. Reflecting on the analysis of two programmes aimed at preventing the death of children by heatstroke, this article illustrates the implications of alternative ‘accident theories’ for the evaluation of safety policies.

Surviving the Anthropocene: How evaluation can contribute to knowledge and better policymaking

We live in the Anthropocene in which human impact on Earth is the dominant force. At the same time, humans are very much part of the ecosystem. This close interdependency is brought home by the COVID-19 pandemic as well as by anthropogenic climate change. Ecosystem health and human health are closely interlinked. Transformational change is required to avoid further catastrophes caused by the three environmental crises that human actions have caused: climate crisis; nature crisis; and pollution and waste crisis. Evaluation can contribute to finding durable solutions based on sound science and experiences from the field, but to do so evaluation must broaden its vision. Theory-based approaches will remain central, but they must be open to the full human and natural systems in which the intervention that is the evaluand operates. Evaluations must also pay attention to unintended consequences of all interventions to the environment, to social and power relations, to women, indigenous peoples and vulnerable groups. I identify three principles for evaluation in the Anthropocene, at the nexus of human and natural systems, and illustrate them using examples from evaluations from the Global Environment Facility: (a) integrating human and natural systems; (b) geographical approaches; and (c) addressing the drivers.

Clarifying realist analytic and interdisciplinary consensus processes in a complex health intervention: A worked example of Judgemental Rationality in action

Judgemental rationality is infrequently referenced within discussions of Realist Evaluations. Judgemental rationality refers to researchers’ capacity to assess which, potential, meanings provide the most credible explanations. In evaluation work, rationale for analysis is provided, though rarely do we see how an evaluator made judgements between competing theories, and which theories were discarded and why. We provide a worked example of the application of judgemental rationality. The Engager intervention offered support to prison leavers with common mental health problems. The data for 24, purposively sampled, participants from the intervention arm of the trial were integrated. Bhaskar’s DREIC, a five-step analytical procedure, was used to transfactually theorise and interrogate the inferences made within, and across, cases. The findings demonstrated that the intervention was more effective when practitioners developed an in-depth understanding of the participant. We recommend that intervention developers look for ways to enhance therapeutic competencies and judgemental rationality in practitioner teams.

How can multi-criteria analysis support deliberative spatial planning? A critical review of methods and participatory frameworks

In spatial planning, the paradigm has shifted from positivist to deliberative approaches. Still, cost–benefit analysis remains the dominant evaluation method. Multi-criteria analysis is arguably more appropriate, as it allows for stakeholder participation. While there are dozens of ever more sophisticated multi-criteria analysis methods, their practicality as real-world learning tools has received little attention. The goal of this article is to assess the suitability of different multi-criteria analysis methods for deliberative planning. It presents a critical review of the logical-mathematical cores of the principal methods but also of the different participatory frameworks within which they can be applied. While mathematically sophisticated methods are valuable in well-defined problems with precise data available, we conclude that in the participatory and politically sensitive stages of the planning process, user-friendly and transparent methods are more appropriate and recommend the development of a method that supports the incremental improvement of design options rather than ranking alternatives.