Using the payback framework to evaluate the outcomes of pilot projects supported by the Georgia Clinical and Translational Science Alliance

Introduction: The Clinical and Translational Science Awards (CTSA) program of the National Center for Advancing Translational Sciences (NCATS) seeks to improve population health by accelerating the translation of scientific discoveries in the laboratory and clinic into practices for the community. CTSAs achieve this goal, in part, through their pilot project programs that fund promising early career investigators and innovative early-stage research projects across the translational research spectrum. However, there have been few reports on individual pilot projects and their impacts on the investigators who receive them and no studies on the long-term impact and outcomes of pilot projects. Methods: The Georgia CTSA funded 183 pilot projects from 2007 to 2015. We used a structured evaluation framework, the payback framework, to document the outcomes of 16 purposefully-selected pilot projects supported by the Georgia CTSA. We used a case study approach including bibliometric analyses of publications associated with the selected projects, document review, and investigator interviews. Results: These pilot projects had positive impact based on outcomes in five “payback categories”: (1) knowledge; (2) research targeting, capacity building, and absorption; (3) policy and product development; (4) health benefits; and (5) broader economic benefits. Conclusions: Results could inform our understanding of the diversity and breadth of outcomes resulting from Georgia CTSA-supported research and provide a framework for evaluating long-term pilot project outcomes across CTSAs.

2218 An application of the payback framework to evaluate the outcomes of pilot projects supported by the Georgia Clinical and Translational Science Alliance from 2007 to 2014

OBJECTIVES/SPECIFIC AIMS: We will use a structured evaluation framework, the payback framework, to document the outcomes of 15 case studies of pilot projects supported by Georgia CTSA from 2007 to 2014. METHODS/STUDY POPULATION: We will use a case study approach including bibliometric analyses of publications associated with the selected projects, document review (e.g., investigator curriculum vitae, biannual project reports) and investigator interviews. RESULTS/ANTICIPATED RESULTS: We will document outcomes in 5 “payback categories”: (1) knowledge, (2) research targeting, capacity building, and absorption, (3) policy and product development, (4) health benefits, and (5) broader economic benefits. DISCUSSION/SIGNIFICANCE OF IMPACT: This study will aid in characterizing the returns resulting from this research funding and identify its strengths and weaknesses. This study will inform our understanding of the diversity and breadth of outcomes resulting from Georgia CTSA-supported research, and the value pilot projects provide to clinical and translational science and the broader community.

Assessing the impact of health technology assessment in the Netherlands

Objectives:Investments in health research should lead to improvements in health and health care. This is also the remit of the main HTA program in the Netherlands. The aims of this study were to assess whether the results of this program have led to such improvements and to analyze how best to assess the impact from health research.Methods:We assessed the impact of individual HTA projects by adapting the “payback framework” developed in the United Kingdom. We conducted dossier reviews and sent a survey to principal investigators of forty-three projects awarded between 2000 and 2003. We then provided an overview of documented output and outcome that was assessed by ten HTA experts using a scoring method. Finally, we conducted five case studies using information from additional dossier review and semistructured key informant interviews.Results:The findings confirm that the payback framework is a useful approach to assess the impact of HTA projects. We identified over 101 peer reviewed papers, more than twenty-five PhDs, citations of research in guidelines (six projects), and implementation of new treatment strategies (eleven projects). The case studies provided greater depth and understanding about the levels of impact that arise and why and how they have been achieved.Conclusions:It is generally too early to determine whether the HTA program led to actual changes in healthcare policy and practice. However, the results can be used as a baseline measurement for future evaluation and can help funding organizations or HTA agencies consider how to assess impact, possibly routinely. This, in turn, could help inform research strategies and justify expenditure for health research.