Does evaluation quality enhance evaluation use?

Background: Evaluations are a useful tool to learn more about the effectiveness of public measures. In the era of evidence-based policymaking, recent research suggests that quality is an important determinant of the utilisation of evaluations. Despite this claim, hardly any empirical study has investigated whether the quality of an evaluation – measured by a meta-evaluation – influences its perceived utilisation by decision makers.Aims and objectives: This article asks how the quality of an evaluation study is related to its perceived use, and investigates the relationship between the quality of an evaluation, assessed through a meta-evaluation, and how the evaluation is perceived and accepted by the parties concerned.Methods: The basis for the empirical analyses were 34 external evaluations, conducted from 2006 to 2014, of upper secondary schools in the canton of Zurich, as well as a standardised survey conducted among 307 representatives of these schools (teachers, administrators, members of quality development teams, and the heads of school oversight commissions).Findings: We conclude that the quality of the evaluation, as assessed in a meta-evaluation, is not particularly associated with the perception of evaluation quality and the perceived use of the evaluation. The perceived quality, however, is related to the perceived impact of an evaluation.Discussion and conclusion: These findings are relevant for evaluation research and practice, since they show that the quality of an evaluation and evaluation use do not necessarily go hand in hand.<br />Key messages<br /><ul><li>Evaluators have to be aware that a systematically assessed quality of an evaluation does not go hand in hand with the perceived quality of that evaluation;</li><br /><li>Evaluators often focus on the instrumental form of evaluation use, but they should not ignore other forms of use and maybe try to maximise these utilisation forms in the design of their evaluation;</li><br /><li>Evaluators should be more active in advising stakeholders when it comes to evaluation use, for example, through policy narratives;</li><br /><li>Evaluators should carefully think about the measurement of evaluation quality and evaluation effects in research on evaluation.</li></ul>

Evaluation Use Theory, Practice, and Future Research: Reflections on the Alkin and King AJE Series

Marvin Alkin and Jean King published three AJE articles on evaluation use over four years, a coherent and comprehensive series covering the historical development of evaluation use, definitions and factors associated with use and misuse, and theories of evaluation use and influence, concluding with assessment of the first 50 years of use research. They conclude with recommendations for future theory development and research on evaluation. I draw a different set of conclusions and pathway forward. Where they seek a common universal operational definition of evaluation use, I propose treating use as a thick sensitizing concept that invites diversity of context-specific meanings. Where they find evaluation use theory inadequate, I argue that it is sufficient for its purpose. Where they seek more development of evaluation-specific utilization theory, I propose drawing on more established and validated theories from social sciences to explain and illuminate evaluation use as occurring in complex dynamic systems.

Explaining use and non-use of policy evaluations in a mature evaluation setting

Governments worldwide have intensified their efforts to institutionalize policy evaluation. Still, also in organizations with high evaluation maturity, the use of evaluations is not self-evident. As mature organizations already meet many of the factors that are commonly seen to foster evaluation use, they constitute an interesting research setting to identify (combinations of) factors that can make a key difference in minimizing research waste. In this article, we present an analysis of the use of evaluations conducted between 2013 and 2016 by the Policy and Operations Evaluation Department (IOB) of the Dutch Ministry of Foreign Affairs, a typical case of relatively high evaluation maturity. Methodologically, we rely on Qualitative Comparative Analysis as an approach that is excellently suited to capture the causal complexity characterizing evaluation use. The analysis provides useful insights on the link between knowledge production and use. We highlight the relevance of engaging policy makers in developing the evaluation design, and fine-tune available evidence as to what is perceived a good timing to organize evaluations. Contrary to existing research, we show that the political salience of an evaluation does not matter much.