Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations

Abstract Researchers and research organizations are under increasing pressure to demonstrate that their work contributes to positive change and helps solve pressing societal challenges. There is a simultaneous trend towards more engaged transdisciplinary research that is complexity-aware and appreciates that change happens through systems transformation, not only through technological innovation. Appropriate evaluation approaches are needed to evidence research impact and generate learning for continual improvement. This is challenging in any research field, but especially for research that crosses disciplinary boundaries and intervenes in complex systems. Moreover, evaluation challenges at the project scale are compounded at the programme scale. The Forest, Trees and Agroforestry (FTA) research programme serves as an example of this evolution in research approach and the resulting evaluation challenges. FTA research is responding to the demand for greater impact with more engaged research following multiple pathways. However, research impact assessment in the CGIAR (Consultative Group on International Agricultural Research) was developed in a technology-centric context where counterfactual approaches of causal inference (experimental and quasi-experimental) predominate. Relying solely on such approaches is inappropriate for evaluating research contributions that target policy and institutional change and systems transformation. Instead, we propose a multifaceted, multi-scale, theory-based evaluation approach. This includes nested project- and programme-scale theories of change (ToCs); research quality assessment; theory-based outcome evaluations to empirically test ToCs and assess policy, institutional, and practice influence; experimental and quasi-experimental impact of FTA-informed ‘large n’ innovations; ex ante impact assessment to estimate potential impacts at scale; and logically and plausibly linking programme-level outcomes to secondary data on development and conservation status.

Engaging with research impact assessment for an environmental science case study

Nature Communications ◽

10.1038/s41467-019-12020-z ◽

2019 ◽

Vol 10 (1) ◽

Cited By ~ 2

Author(s):

Kirstie A. Fryirs ◽

Gary J. Brierley ◽

Thom Dixon

Keyword(s):

Impact Assessment ◽

Environmental Science ◽

Academic Research ◽

Research Impact ◽

Research Field ◽

Rating Systems ◽

Impact Pathway ◽

Abstract Impact assessment is embedded in many national and international research rating systems. Most applications use the Research Impact Pathway to track inputs, activities, outputs and outcomes of an invention or initiative to assess impact beyond scholarly contributions to an academic research field (i.e., benefits to environment, society, economy and culture). Existing approaches emphasise easy to attribute ‘hard’ impacts, and fail to include a range of ‘soft’ impacts that are less easy to attribute, yet are often a dominant part of the impact mix. Here, we develop an inclusive 3-part impact mapping approach. We demonstrate its application using an environmental initiative.

Assistive tools for disability arts: collaborative experiences in working with disabled artists and stakeholders

Journal of Assistive Technologies ◽

10.1108/jat-12-2015-0034 ◽

2016 ◽

Vol 10 (2) ◽

pp. 121-129 ◽

Cited By ~ 6

Author(s):

Chris Creed

Keyword(s):

Special Needs ◽

Research Impact ◽

Research Process ◽

Research Approach ◽

Visual Artists ◽

Content Type ◽

British Council ◽

Artistic Work ◽

Conducting Research ◽

Purpose – The purpose of this paper is to provide a review of the experiences in working collaboratively with physically impaired visual artists and other stakeholders (e.g. disability arts organisations, charities, personal assistants, special needs colleges, assistive technologists, etc.) to explore the potential of digital assistive tools to support and transform practice. Design/methodology/approach – The authors strategically identified key organisations as project partners including Disability Arts Shropshire, Arts Council England, the British Council, SCOPE, and National Star College (a large special needs college). This multi-disciplinary team worked together to develop relationships with disabled artists and to collaboratively influence the research focus around investigating the current practice of physically impaired artists and the impact of digital technologies on artistic work. Findings – The collaborations with disabled artists and stakeholders throughout the research process have enriched the project, broadened and deepened research impact, and enabled a firsthand understanding of the issues around using assistive technology for artistic work. Artists and stakeholders have become pro-active collaborators and advocates for the project as opposed to being used only for evaluation purposes. A flexible research approach was crucial in helping to facilitate research studies and enhance impact of the work. Originality/value – This paper is the first to discuss experiences in working with physically impaired visual artists – including the benefits of a collaborative approach and the considerations that must be made when conducting research in this area. The observations are also relevant to researchers working with disabled participants in other fields.

Defining and evaluating the impact of cross-disciplinary conservation research

Environmental Conservation ◽

10.1017/s0376892910000792 ◽

2010 ◽

Vol 37 (4) ◽

pp. 442-450 ◽

Cited By ~ 24

Author(s):

ANNA C. EVELY ◽

IOAN FAZEY ◽

XAVIER LAMBIN ◽

EMILY LAMBERT ◽

SARAH ALLEN ◽

...

Keyword(s):

Research Impact ◽

Environmental Problems ◽

Research Quality ◽

Conservation Research ◽

Definition Of ◽

SUMMARYCross-disciplinary research is advocated as a way of improving understanding of the complexity of environmental problems; cross-disciplinary projects, centres and academic institutes have increased. However, there is confusion over the nature of cross-disciplinary research. Through review of papers defining themselves as cross-disciplinary that aim to address conservation problems, and by standardizing the definition of cross-disciplinary research, it is possible to evaluate the potential research impact on peers and practitioners. When papers were reclassified by authors, those reclassified as transdisciplinary were perceived to have a greater impact on practitioners, and those reclassified as non cross-disciplinary had the greatest impact on colleagues. Having clear definitions for types of cross-disciplinary research would help establish a firm foundation, not only for improving research quality, but also for evaluating research impact. While the number of cross-disciplinary studies is increasing, cross-disciplinary research falls short of integrating disciplinary methods in much depth and does not have much impact on participants outside of academia.

Introduction: critical reflections on research impact

The Impact Agenda ◽

10.1332/policypress/9781447339854.003.0001 ◽

2020 ◽

pp. 1-12

Author(s):

Katherine E. Smith ◽

Justyna Bandola-Gill ◽

Nasar Meer ◽

Ellen Stewart ◽

Richard Watermeyer

Keyword(s):

Impact Assessment ◽

Empirical Investigation ◽

Research Impact ◽

The Uk ◽

The Impact ◽

Critical Reflections

This chapter briefly explains what we mean by ‘the impact agenda’ and what the UK approach to research impact assessment involves. This chapter also makes the case for why an empirical investigation of the recent changes associated with research impact assessment is required and provides key definitions and an overview of the rest of the book.

A Study on Methodologies for Research Impact Assessment: Responses of the UK Research Councils to the Warry Report

Industry and Higher Education ◽

10.5367/ihe.2010.0006 ◽

2010 ◽

Vol 24 (5) ◽

pp. 393-397 ◽

Cited By ~ 1

Author(s):

André Luiz de Campos

Keyword(s):

Impact Assessment ◽

Research Funding ◽

Research Impact ◽

Policy Makers ◽

Impact Studies ◽

The Uk ◽

Research Councils ◽

The experience of the UK Research Councils in assessing the impacts of their research funding is discussed, including a report on the findings of research which reviewed the impact studies implemented by the Research Councils. The response of the Councils to the challenge of demonstrating the impacts of their funding and the main methodologies used are presented and the implications of both for the Research Councils and policy makers elsewhere are outlined.

The Impact of Regular Research Field Hearings (RRFH) in On-farm Trials in Northeast Brazil

Experimental Agriculture ◽

10.1017/s0014479700016136 ◽

1988 ◽

Vol 24 (3) ◽

pp. 281-288 ◽

Cited By ~ 4

Author(s):

Greg Baker ◽

Hendrik C. Knipscheer ◽

Jose De Souza Neto

Keyword(s):

Agricultural Research ◽

Farming Systems ◽

Research Field ◽

Performance Criteria ◽

Northeast Brazil ◽

Test Results ◽

Systems Research ◽

On Farm ◽

The Impact ◽

Farming Systems Research

SUMMARYThe study of livestock in farming systems research presents several challenges. Factors such as the small number of observations, high statistical variability in measurements, and the difficulty in valuing non-market inputs and outputs make it difficult to evaluate on-farm research. Regular Research Field Hearings (RRFH) can improve communication between scientists and farmers, thereby increasing mutual understanding, willingness to collaborate in evaluation, potential technologies, and also improving test results. Of three groups of sheep and goat farmers involved in on-farm testing in northeast Brazil, the group which participated in RRFH experienced a significantly higher level of success with the technology as measured according to performance criteria such as weight gains and farmers' perceptions of their animals' performance. The results confirm the importance of farmers' participation in agricultural research.

The significance of impact in real estate research publications

Journal of Property Investment and Finance ◽

10.1108/jpif-04-2020-0034 ◽

2021 ◽

Vol ahead-of-print (ahead-of-print) ◽

Author(s):

Graeme Newell ◽

Muhammad Jufri Marzuki ◽

Elaine Worzala ◽

Alastair Adair ◽

Martin Hoesli ◽

...

Keyword(s):

Real Estate ◽

Urban Policy ◽

Research Impact ◽

High Impact ◽

Early Career ◽

Research Quality ◽

Content Type ◽

Publish Or Perish ◽

Real Estate Research ◽

PurposeResearch impact has taken on increased importance at both a micro- and macro-level and is a key factor today in shaping the careers of real estate researchers. This has seen a range of research impact metrics become global benchmarks when assessing research impact at the individual academic level and journal level. Whilst recognising the limitations of research impact metrics, this paper uses these research impact metrics to identify the leading research impact researchers in real estate, as well as the leading real estate journals in the real estate impact space. The nexus between research quality and research impact is also articulated. As well as focusing on research quality, strategies are identified for the effective incorporation of research impact into a real estate researcher's agenda to assist their research careers; particularly for Early Career Researchers in real estate.Design/methodology/approachThe research impact profile of over 150 real estate researchers and 22 real estate journals was assessed using Google Scholar and Publish or Perish. Using the research impact metrics of the h-index, total citations and i10, the leading high impact real estate researchers as well as the high impact real estate journals are identified.FindingsBased in these research impact metrics, the leading real estate researchers in impactful real estate research are identified. Whilst being US focused, there is clear evidence of increasing roles by ERES, AsRES and PRRES players. The leading real estate journals in the impact space are identified, including both real estate-specific journals and the broader planning/urban policy journals, as well as being beyond just the standard US real estate journals. Researcher career strategies are also identified to see both research quality and research impact included as balanced elements in a real estate researcher's career strategy.Practical implicationsWith research impact playing an increased role in all real estate researchers' careers, the insights from this paper provide strong empirical evidence for effective strategies to expand the focus on the impact of their real estate research agendas. This sees a balanced strategy around both research quality and research impact as the most effective strategy for real estate researchers to achieve their research career goals.Originality/valueResearch impact has taken on increased importance globally and is an important factor in shaping real estate researchers' careers. Using research impact metrics, this is the first paper to rigorously and empirically identify the leading research impact players and journals in real estate, as well as identifying strategies for the more effective inclusion of impact in real estate researchers' agendas.

GEOINFO: impactos sociais do repositório de dados de pesquisa da EMBRAPA

Páginas a&b Arquivos & Bibliotecas ◽

10.21747/21836671/pagnesppk17 ◽

2020 ◽

pp. 145-149

Author(s):

Daniela Maciel Pinto ◽

Geraldo Stachetti Rodrigues ◽

Gustavo Spadotti Amaral Castro ◽

Gisele Vilela Freitas ◽

Angelo Mansur Mendes ◽

...

Keyword(s):

Impact Assessment ◽

Information And Communication Technologies ◽

Information Science ◽

Science Studies ◽

Agricultural Research ◽

Research Data ◽

Social Dimension ◽

Assessment Process ◽

Data Repositories ◽

GeoInfo is a repository that provides the spatial research data generated by the Brazilian Agricultural Research Corporation (EMBRAPA) to understand the dynamics of agriculture in Brazilian territory. Considering the efforts required for the implantation and institutionalization of the repository, the absence of Information Science studies related to the investigation of the impact of the uses made from the data and information made available in research data repositories, as well as the impact assessment process used by EMBRAPA since 1989, it is opportune to investigate the impact of the repository for its target audience, that is, geoscience specialists. Thus, the objective of this work is to present the results of an impact assessment carried out with GeoInfo, based on the “Ambitec-TICs: Module of criteria and impact indicators for Information and Communication Technologies”, specifically for the social dimension, existing at Ambitec-TICs

The Impact Agenda

10.1332/policypress/9781447339854.001.0001 ◽

2020 ◽

Author(s):

Katherine Smith ◽

Justyna Bandola-Gill ◽

Nasar Meer ◽

Ellen Stewart ◽

Richard Watermeyer

Keyword(s):

Impact Assessment ◽

Academic Research ◽

Research Impact ◽

Empirical Examination ◽

Societal Benefit ◽

Wide Range ◽

Academic Practices ◽

The Uk ◽

As international interest in promoting and assessing the impact of research grows, this book examines the ensuing controversies, consequences and challenges. It places a particular emphasis on learning from experiences in the UK, since this is the country at the forefront of a range of new approaches to incentivising, monitoring and rewarding research impact achievements. The book aims to understand the origins and rationale for these changes and to critically assess their consequences for academic practice. Combining a review of existing literature with a range of new qualitative data (from interviews, focus groups and documentary analysis), The Impact Agenda is unique in providing a comprehensive, cross-disciplinary empirical examination of the ways in which various forms of research impact assessment are shaping academic practices. Although the primary focus of the book is on the UK, the book also considers the different approaches that other countries with an interest in research impact are taking (notably Australia, Canada and the Netherlands). While noting the benefits that the increasing emphasis on outward facing work is bringing, the book draws attention to a wide range of challenges and controversies associated with research impact assessment and, in particular, with the UK’s chosen approach. It concludes by using the insights in the book to propose an alternative, more theoretically robust approach to incentivising and rewarding efforts to undertake and use academic research for societal benefit.