Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic

The COVID-19 pandemic required large-scale service delivery changes for government, and provided the opportunity for evaluators to step up and support decision makers to understand the impact of these changes. Rapid evaluation methods (REM) provide a pragmatic approach for generating timely information for evidence-based policy and decision-making. Grounded in developmental and utilisation-focused evaluation theory, REM incorporates a team-based, mixed methods design, executed over a 6–8-week period. Customised rubrics were used to rigorously assess effectiveness and scalability of practice changes to inform COVID-19 response planning. REM is an alternative approach to full-scale evaluation models frequently implemented to assess policies and programs. Adapted use of REM suggests that meaningful insights can be gained through use of smaller scale evaluations. This article shares lessons learned from a novel rapid evaluation method applied in the context of the COVID-19 pandemic. The rapid evaluation approach was implemented to provide real-time insights and evaluative conclusions for 15 program and practice adaptations across Victorian health and human service settings. The article shares insights about the practical applicability of balancing rigour and timeliness when implementing a rapid evaluation, and strengths and limitations of working within a fast-paced evaluation framework. Findings can inform evaluative practice in resource and time-limited settings.

Evaluator perspective: An interview with Nicole Tujague

Nicole tells us her story of coming to a decision to work more closely in the Indigenous space. She embarked on a journey of self-discovery which resulted in not only understanding herself but also her own people. One of her important discoveries was finding the Australian Evaluation Society (AES) and meeting people of like minds. She questioned projects where measurement is determined by the funder, not by the Indigenous people who are being impacted by the evaluation, and on the programmes which affect their lives. She has also realised that it is not enough to do evaluation in a culturally safe way, but that evaluation should be culturally safe and trauma informed. You need to understand the trauma histories of people and communities and that knowledge needs to be embedded in your methodology. Without that knowledge, you cannot expect to understand the community and its issues or produce outcomes that are healing.

Cross-cultural realist interviews: An integration of the realist interview and cross-cultural qualitative research methods

Realist interviews are a data collection method used in realist evaluations. There is little available guidance for realist interviewing in cross-cultural contexts. Few published realist evaluations have included cross-cultural interviews, providing limited analyses of the cross-cultural application of realist methodology. This study integrated realist and cross-cultural qualitative methods in a realist evaluation of an Australian doula support program. The interviews were conducted with Arabic speaking clients of the program. The process included collaboration with a bicultural researcher, philosophically situating the study for methodologically coherent integration, bicultural review of the appropriateness of realist ‘how’ and ‘why’ questions, decisions about language translation and interpretation, pilot interviews, and co-facilitation of the interviews. Integration of the methods was feasible and valuable. This study may support other realist evaluators to give voice to people from culturally diverse groups, in a manner that is culturally safe, methodologically coherent and rigorous, and that produces trustworthy results.

Conversations that Count: Lessons From Evaluating a Men’s Digital Mental Health Response During COVID-19

This paper details disruption and innovation in digital evaluation practice at Movember, as a result of the COVID-19 pandemic. The paper examines a men’s digital health intervention (DHI) – Movember Conversations – and the product pivot that was necessary to ensure it could respond to the pandemic. The paper focuses on the implications of the pivot for the evaluation and how the evaluation was adapted to the COVID-19 exigencies. It details the redesign of the evaluation to ensure methods wrapped around the modified product and could deliver real-time, practical insights. The paper seeks to fill knowledge gaps in the DHI evaluation space and outlines four key principles that support evaluation re-design in an agile setting. These include a user-centred approach to evaluation design, proportionate data collection, mixed (and flexible) methodologies, and agile evaluation reporting. The paper concludes with key lessons and reflections from the evaluators about what worked at Movember, to support other evaluators planning digital evaluations.

Enhancing the inclusion of gender and sexually diverse populations in evaluation: Reflections grounded in practice

Evaluators are committed to practice that is high quality and conducted ethically. Despite this, gender and sexually diverse populations are not always adequately considered in evaluation practice. Ensuring fuller inclusion of gender and sexually diverse people is required to give effect to human rights obligations and to enable comparatively poorer wellbeing outcomes for these groups to be addressed. Based on our experience conducting evaluation (and research) with both general populations and gender and sexually diverse populations, we suggest a need to build inclusive practice among evaluators. To guide inclusive evaluation practice, we outline three domains for consideration – terminology and language, processes of research inclusion and implications of inclusion. These are not offered as a checklist but as a way to encourage reflexive practice among evaluators. Given evaluators are often concerned with promoting equity and social justice, we are hopeful that actions taken by evaluators can enhance the inclusion of gender and sexually diverse people in evaluation activities and contribute to better wellbeing outcomes for them.

COVID-19 concerns, health services utilisation and social support among Western Australians with diabetes during the pandemic

People with diabetes face increased risk of serious COVID-19 complications, making self-care for optimal metabolic management crucial. However, the pandemic has reduced access to routine care among people with diabetes. The pandemic can also elicit distress, which can impact diabetes self-management and health. To understand the impact of COVID-19 on Western Australians with diabetes, we conducted an evaluation involving an online survey of consumers of diabetes health services and an analysis of routine program data (i.e. service utilisation/program attendance). Survey respondents were concerned about contracting COVID-19, many intended to change the way they utilised health services and many indicated they would continue to socially isolate. Utilisation of digital/telephone services peaked between April and June 2020. Despite the concerns indicated, a participation resurgence was observed upon resumption of face-to-face programs. Continued access to diabetes programs via multiple modes of delivery is critical to support optimal self-care and mitigate COVID-19 risks, distress and social isolation. This timely and pragmatic assessment of consumer beliefs synthesised with routinely collected evaluation data represents an agile approach to evaluation through an emerging public health crisis. The findings helped to ensure optimal service delivery to meet the needs of this priority population throughout the pandemic.