Theory-based evaluations: Framing the existence of a new theory in evaluation and the rise of the 5th generation

Evaluation ◽  
2018 ◽  
Vol 24 (2) ◽  
pp. 153-168 ◽  
Author(s):  
Astrid Brousselle ◽  
Jean-Marie Buregeya
Keyword(s):  
Critical Realism ◽  
Complex Interventions ◽  
Realist Evaluation ◽  
Fifth Generation ◽  
Contribution Analysis ◽  
Generation Theory ◽  
Similarities And Differences ◽  
Explanation Generation ◽  
Design And Application

In this article we defend the idea that theory-based evaluations—contribution analysis, logic analysis, and realist evaluation—are complementary components of a new theory in evaluation. We also posit that we are currently observing the emergence of a fifth generation in evaluation: the explanation generation. Theory-based evaluations have featured prominently in the discourse of evaluators since the mid-1980s. They have developed mainly in response to the need for evaluation of complex interventions. In this article we analyze certain approaches that have matured in their design and application. We use the framework of Shadish et al. to analyze the ontological, epistemological, and methodological foundations of various theory-based approaches in evaluation to appraise their similarities and differences. We observe that all these approaches are grounded in critical realism. Similarities seen in their ontological, epistemological, and methodological positionings, as well as their complementarity in terms of the evaluative questions they address, suggest we may be observing the consolidation of a new theory in evaluation and the emergence of a fifth generation.

Health and illness research: value-free or value-laden?

2021 ◽  
pp. 95-126
Author(s):  
Priscilla Alderson
Keyword(s):  
Health Research ◽  
Moral Realism ◽  
Critical Realism ◽  
Realist Evaluation ◽  
Ethical Naturalism ◽  
Ethical Concerns ◽  
Research Paradigms ◽  
Health And Illness ◽  
Health Related

This chapter asks whether health research can and should be value-free. It questions whether facts can be separated from values, and then considers: health-related rights; dignity in healthcare; truth, trust and consent; values and ethics in health research. The chapter also looks at: what critical realism can add on important ethical concerns that are missing or under-examined in other paradigms; health research paradigms and ethics; ethical naturalism and moral realism; learning from other major theorists: advocacy on many levels and, finally, the chapter compares the approaches to values in realist evaluation and in critical realism. In the detailed example, Graham Scambler combines critical realism with Habermas’s theories.

9 Using what people value to develop new interventions in palliative care: a multilevel level review approach

2018 ◽  
Vol 8 (3) ◽  
pp. 363.1-363
Author(s):  
B Candy ◽  
M Armstrong ◽  
R Amey ◽  
J Booth ◽  
K Flemming ◽  
...  
Keyword(s):  
Palliative Care ◽  
Complementary Therapies ◽  
Complex Interventions ◽  
Trial Data ◽  
Qualitative Studies ◽  
The Matrix ◽  
Intervention Content ◽  
Novel Approaches ◽  
Similarities And Differences ◽  
Meta Analyses

IntroductionComplex interventions are common in palliative care (PC) but are difficult to evaluate. Complementary therapies (CT) are one type of widely-used complex intervention for which there is inconclusive evidence. No systematic review has been conducted in PC of trials or of qualitative studies of patients’ views of CTs. There are novel approaches with established exemplars of using both types of reviews to help develop more clinically appropriate interventions. These approaches are in their infancy in PC research and have much to offer the specialism.AimsUsing our on-going review on CT we present as an exemplar in PC an approach to draw together the findings of trials and qualitative studies in a data table (matrix) to contrast what patients value and want with how the intervention is tested.MethodsWe sought trials on the effectiveness of CT and qualitative studies on patients’ perspectives about these therapies. Our primary outcomes for trials included anxiety. Eight databases were searched in 2017. Citations and full-text papers were reviewed to identify relevant studies. Meta-analyses pooled trial data where appropriate and a thematic synthesis is being undertaken to understand patient experience. These findings will be combined in a matrix to explore similarities and differences.Results19 trials and five qualitative studies were included. Data analysis and development of the matrix which includes intervention content and patients’ needs is currently underway. We will present the final matrix framework.ConclusionsOur approach demonstrates a potential way in PC to enhance development of practice appropriate complex interventions.

scholarly journals Applying Meta-Theory to Qualitative and Mixed-Methods Research

2018 ◽  
Vol 17 (1) ◽  
pp. 160940691879004 ◽  
Author(s):  
Saleema Allana ◽  
Alexander Clark
Keyword(s):  
Mixed Methods ◽  
Mixed Methods Research ◽  
Systems Approach ◽  
Critical Realism ◽  
Open Systems ◽  
Complex Interventions ◽  
Practice Research ◽  
Research And Practice ◽  
Management Interventions ◽  
Meta Theory

Meta-theory refers to broad perspectives, which make claims regarding the nature of reality. Meta-theories philosophically underpin research and practice. Despite this centrality of meta-theory to research and practice, research studies seldom have a strong articulated philosophical basis. There are persuasive philosophical arguments for invoking meta-theory in qualitative and mixed-methods research. We argue that selecting and applying a particular meta-theory is a matter of personal expression and historicity. In this article, we describe the meta-theory of critical realism (CR), which underpins our research around complex heart failure disease management interventions. CR posits that reality is mind independent and views this reality via a stratified ontology. Its explanatory focus, generative logic, multifactorial and open systems approach, and its openness to a variety of methods make it a viable meta-theory for research in a variety of disciplines, utilizing qualitative, quantitative, and mixed methods. CR hermeneutics, ethnographies, grounded theories, mixed-methods studies, and critical realist reviews follow the meta-theoretical assumptions of CR; these are extremely useful in exploring complex interventions holistically, including their components, contexts, and mechanisms.

Rethinking theories: the basis of practical research and problems with paradigms

2021 ◽  
pp. 11-40
Author(s):  
Priscilla Alderson
Keyword(s):  
Health Research ◽  
Critical Realism ◽  
Care Service ◽  
Realist Evaluation ◽  
Social Health ◽  
Community Based ◽  
Practical Research ◽  
Global Problems ◽  
Health And Illness

Chapter 1 reviews: social and biomedical influences through the centuries; positivist health research and then problems with it; interpretive research and some associated problems; differences between positivism and interpretivism; realist evaluation compared and contrasted with critical realism and discourse analysis. The detailed example is an evaluation of a community-based integrated care service. The chapter ends with questioning the present unhealthy state of social health research, which undermines the hope that sociology can be generally respected, convincing and useful in helping to resolve serious global problems of health and illness. This prepares for future chapters that show how critical realism draws on the strengths and overcomes the limitations of these paradigms by combining them into larger analysis.

scholarly journals Realistic Evaluation as both a science and as a methodology

2019 ◽  
Vol 18 (4) ◽  
pp. 228-230
Author(s):  
Paul Linsley
Keyword(s):  
Social Sciences ◽  
Mixed Methods ◽  
Critical Realism ◽  
Evaluation Study ◽  
Realist Evaluation ◽  
Realistic Evaluation

AbstractThis paper examines an increasingly used methodology developed by Pawson and Tilley [1] that they termed, ‘Realistic Evaluation’. The paper provides an overview of methodology as well as exploring its underpinning philosophy, that of critical realism. The paper suggests that Realistic Evaluation provides a useful means of exploring the health and social sciences through the use of mixed methods and concludes with how a researcher might go about constructing a Realist Evaluation study.

scholarly journals Quality and reporting standards, resources, training materials and information for realist evaluation: the RAMESES II project

2017 ◽  
Vol 5 (28) ◽  
pp. 1-108 ◽  
Author(s):  
Geoff Wong ◽  
Gill Westhorp ◽  
Joanne Greenhalgh ◽  
Ana Manzano ◽  
Justin Jagosh ◽  
...  
Keyword(s):  
Literature Review ◽  
Complex Interventions ◽  
Research Programme ◽  
Quality Standards ◽  
Research Capacity ◽  
Realist Evaluation ◽  
Delphi Panel ◽  
Reporting Standards ◽  
Training Materials ◽  
And Training

BackgroundMany of the problems confronting policy- and decision-makers, evaluators and researchers today are complex, as are the interventions designed to tackle them. Their success depends both on individuals’ responses and on the wider context of people’s lives. Realist evaluation tries to make sense of these complex interventions. It is a form of theory-driven evaluation, based on realist philosophy, that aims to understand why these complex interventions work, how, for whom, in what context and to what extent.ObjectivesOur objectives were to develop (a) quality standards, (b) reporting standards, (c) resources and training materials, (d) information and resources for patients and other lay participants and (e) to build research capacity among those interested in realist evaluation.MethodsTo develop the quality and reporting standards, we undertook a thematic review of the literature, supplemented by our content expertise and feedback from presentations and workshops. We synthesised findings into briefing materials for realist evaluations for the Delphi panel (a structured method using experts to develop consensus). To develop our resources and training materials, we drew on our experience in developing and delivering education materials, feedback from the Delphi panel, the RAMESES JISCMail e-mail list, training workshops and feedback from training sessions. To develop information and resources for patients and other lay participants in realist evaluation, we convened a group consisting of patients and the public. We built research capacity by running workshops and training sessions.ResultsOur literature review identified 152 realist evaluations, and when 37 of these had been analysed we were able to develop our briefing materials for the Delphi panel. The Delphi panel comprised 35 members from 27 organisations across six countries and five disciplines. Within three rounds, the panels had reached a consensus on 20 key reporting standards. The quality standards consist of eight criteria for realist evaluations. We developed resources and training materials for 15 theoretical and methodological topics. All resources are available online (www.ramesesproject.org). We provided methodological support to 17 projects and presentations or workshops to help build research capacity in realist evaluations to 29 organisations. Finally, we produced a generic patient information leaflet for lay participants in realist evaluations.LimitationsOur project had ambitious goals that created a substantial workload, leading to the need to prioritise objectives. For example, we truncated the literature review and focused on standards and training material development.ConclusionsAlthough realist evaluation holds much promise, misunderstandings and misapplications of it are common. We hope that our project’s outputs and activities will help to address these problems. Our resources are the start of an iterative journey of refinement and development of better resources for realist evaluations. The RAMESES II project seeks not to produce the last word on these issues, but to capture current expertise and establish an agreed state of the science. Much methodological development is needed in realist evaluation but this can take place only if there is a sufficient pool of highly skilled realist evaluators. Capacity building is the next key step in realist evaluation.FundingThe National Institute for Health Research Health Services and Delivery Research programme.

Using contribution analysis to evaluate large-scale, transformation change processes

Evaluation ◽  
2020 ◽  
Vol 26 (2) ◽  
pp. 227-245
Author(s):  
Kerstin Junge ◽  
Joe Cullen ◽  
Giorgia Iacopini
Keyword(s):  
Large Scale ◽  
Complex Interventions ◽  
Transformational Change ◽  
Scale Transformation ◽  
Change Processes ◽  
National Scale ◽  
Complex Change ◽  
Contribution Analysis

This article explores the usefulness of Contribution Analysis to understand and assess the outcomes and impacts of emergent, large-scale and transformational change processes. As these take place over long periods of time and at national if not supra-national scale, they amplify some of the key challenges complex interventions pose to evaluators, most notably that of the emergence of activities and outcomes. While Contribution Analysis is discussed in the literature as useful for the evaluation of complex change processes, it is not usually recommended for experimental and emergent interventions. This article draws on the authors’ experience of using Contribution Analysis in one such case and explores the relative merits of the method for large-scale/transformational change interventions generally.

Structure and agency: making connections

2021 ◽  
pp. 65-94
Author(s):  
Priscilla Alderson
Keyword(s):  
Mental Health ◽  
Traumatic Brain Injury ◽  
Mental Health Problems ◽  
Critical Realism ◽  
Realist Evaluation ◽  
Social Structures ◽  
Injury Rehabilitation ◽  
Structure And Agency ◽  
Critical Realist ◽  
Health And Illness

This chapter continues to summarise different theories and methods in mainstream health and illness research in order to compare them to critical realism and show what it might add. Positivist, interpretive and postmodern approaches to human agency within structures are considered. There are also sections on: six features of critical realism; structure, agency and culture; four types of social structures; a comparison of realist evaluation and critical realism theories of structure and agency; the structure-agency dialectic, and Archer’s theory of internal conversations. The detailed examples of applying critical realist concepts are of prison leavers with mental health problems, and nurses working on traumatic brain injury rehabilitation. Throughout, the chapter critically considers problems posed for health researchers, to see how critical realist research theories of structure and agency can serve their work.

scholarly journals The Building Blocks of Implementation Frameworks and Models in Primary Care: A Narrative Review

2021 ◽  
Vol 9 ◽  
Author(s):  
Ine Huybrechts ◽  
Anja Declercq ◽  
Emily Verté ◽  
Peter Raeymaeckers ◽  
Sibyl Anthierens
Keyword(s):  
Primary Care ◽  
Complex Interventions ◽  
Implementation Strategies ◽  
Building Blocks ◽  
Narrative Review ◽  
Process Models ◽  
Implementation Framework ◽  
The Core ◽  
Core Components ◽  
Similarities And Differences

Background: Our aim is to identify the core building blocks of existing implementation frameworks and models, which can be used as a basis to further develop a framework for the implementation of complex interventions within primary care practices. Within the field of implementation science, various frameworks, and models exist to support the uptake of research findings and evidence-based practices. However, these frameworks and models often are not sufficiently actionable or targeted for use by intervention designers. The objective of this research is to map the similarities and differences of various frameworks and models, in order to find key constructs that form the foundation of an implementation framework or model that is to be developed.Methods: A narrative review was conducted, searching for papers that describe a framework or model for implementation by means of various search terms, and a snowball approach. The core phases, components, or other elements of each framework or model are extracted and listed. We analyze the similarities and differences between the frameworks and models and elaborate on their core building blocks. These core building blocks form the basis of an overarching model that we will develop based upon this review and put into practice.Results: A total of 28 implementation frameworks and models are included in our analysis. Throughout 15 process models, a total of 67 phases, steps or requirements are extracted and throughout 17 determinant frameworks a total of 90 components, constructs, or elements are extracted and listed into an Excel file. They are bundled and categorized using NVivo 12© and synthesized into three core phases and three core components of an implementation process as common elements of most implementation frameworks or models. The core phases are a development phase, a translation phase, and a sustainment phase. The core components are the intended change, the context, and implementation strategies.Discussion: We have identified the core building blocks of an implementation framework or model, which can be synthesized in three core phases and three core components. These will be the foundation for further research that aims to develop a new model that will guide and support intervention designers to develop and implement complex interventions, while taking account contextual factors.

Sign in / Sign up

Export Citation Format

Share Document