Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System

This paper focuses on the application of systems thinking concepts to evaluate systems. The terms systems and systems thinking concepts are first defined. The use of systems thinking concepts in program and system evaluation are then highlighted. It is noted that while there are methods available to assist evaluation practitioners in applying systems thinking concepts to program evaluation, there is a need for similar guidance in applying systems thinking concepts to evaluating systems. System Evaluation Theory (SET) is then reviewed as one evaluation theory designed to apply systems thinking concepts to evaluating systems. A case illustration is presented to help teach practitioners how to apply SET’s three steps. The discussion focuses on comparing the differences between evaluation questions answered by applying systems thinking concepts versus those using program logic models.

Download Full-text

Evaluating Youth Programs: An Overview of Practitioner Led Evaluations

Journal of Youth Development ◽

10.5195/jyd.2010.236 ◽

2010 ◽

Vol 5 (1) ◽

pp. 112-117

Author(s):

Mat D. Duerden ◽

Peter A. Witt

Keyword(s):

Program Evaluation ◽

Youth Programs ◽

Positive Development ◽

Positive Outcomes ◽

Program Logic ◽

Evaluation Strategy ◽

Logic Models ◽

Different Types ◽

Key Issues

Youth programs are prime contexts for the intentional facilitation of positive development. However, not all youth programs achieve positive outcomes equally. In order to promote the identification and dissemination of the characteristics and processes of effective youth programs, increased focus needs to be given to program evaluation. This article briefly reviews the main tenets of evaluation science in order to provide practitioners a roadmap for conducting their own evaluations. This includes an overview of different types of evaluations and key issues to consider when constructing an evaluation strategy such as targeting outcomes and developing program logic models.

Download Full-text

Defining Systems to Evaluate System Efficiency and Effectiveness

Evaluation Journal of Australasia ◽

10.1177/1035719x1701700302 ◽

2017 ◽

Vol 17 (3) ◽

pp. 4-13 ◽

Cited By ~ 2

Author(s):

Ralph Renger ◽

Jirina Foltysova ◽

Jessica Renger ◽

Wayne Booze

Keyword(s):

Systems Thinking ◽

Lessons Learned ◽

Cardiac Care ◽

System Efficiency ◽

System Boundaries ◽

Evaluation Theory ◽

Theoretical Rationale ◽

Efficiency And Effectiveness ◽

Care Systems ◽

The Relationship

This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no literature describing how to define these system elements. Using an example from the evaluation of several United States cardiac care systems, the paper shares a number of methods used to define the system boundaries, components, and relationships. The paper describes how each of these elements informs the evaluation of step two of SET—evaluating system efficiency. The discussion shares lessons learned, and notes the relationship between methods used in system and program evaluation.

Download Full-text

Use of Program Logic Models in the Southern Rural Access Program Evaluation

The Journal of Rural Health ◽

10.1111/j.1748-0361.2003.tb00646.x ◽

2003 ◽

Vol 19 ◽

pp. 308-313 ◽

Cited By ~ 4

Author(s):

Donald Pathman ◽

Samruddhi Thaker ◽

Thomas C. Ricketts ◽

Jennifer B. Albright

Keyword(s):

Program Evaluation ◽

Program Logic ◽

Logic Models ◽

Rural Access ◽

Access Program

Download Full-text

Use of Program Logic Models in the Southern Rural Access Program Evaluation

The Journal of Rural Health ◽

10.1111/j.1748-0361.2003.tb01049.x ◽

2003 ◽

Vol 19 (5) ◽

pp. 308-313 ◽

Cited By ~ 1

Author(s):

Donald Pathman ◽

Samruddhi Thaker ◽

Thomas C. Ricketts ◽

Jennifer B. Albright

Keyword(s):

Program Evaluation ◽

Program Logic ◽

Logic Models ◽

Rural Access ◽

Access Program

Download Full-text

Illustrating the Evaluation of System Feedback Mechanisms Using System Evaluation Theory (SET)

Evaluation Journal of Australasia ◽

10.1177/1035719x1601600403 ◽

2016 ◽

Vol 16 (4) ◽

pp. 14-20 ◽

Cited By ~ 4

Author(s):

Ralph Renger

Keyword(s):

Evaluation Criteria ◽

System Evaluation ◽

Cardiac Care ◽

System Efficiency ◽

Evaluation Theory ◽

Feedback Mechanisms ◽

Key Factor ◽

Response Systems ◽

Post Hoc ◽

Care Response

This article describes how system evaluation theory (SET) guided the evaluation of cardiac care response systems efficiency in seven rural United States. Specifically, the article focuses on the approach and methods used to evaluate system feedback mechanisms; one key factor affecting system efficiency. Mixed methods were applied to evaluate five criteria of system feedback efficiency: frequency, timeliness, credibility, specificity, and relevance. Examples from the cardiac care response system evaluation are used to illustrate each of the evaluation criteria. The discussion contrasts the role of the evaluator in system versus program evaluation, notes the post-hoc support of SET system attributes in affecting system efficiency, and offers additional consideration in evaluating system feedback mechanisms.

Download Full-text

What’s Going On? Developing Program Theory for Evaluation

Journal of Youth Development ◽

10.5195/jyd.2010.222 ◽

2010 ◽

Vol 5 (2) ◽

pp. 71-82

Author(s):

Mary E. Arnold ◽

Brooke D. Nott

Keyword(s):

Program Evaluation ◽

Large Scale ◽

Logic Model ◽

Program Theory ◽

Program Logic ◽

Scale Evaluation ◽

Program Success ◽

Logic Models ◽

Program Policy ◽

Program Logic Model

This paper outlines the process undertaken to develop a program logic model into a program theory that could be tested through a large scale evaluation. Logic models are widely promoted to establish program intention and outcomes, thus setting the stage for effective program evaluation. However, the causal links in program logic models are often assumptive in nature, and lack the specificity of conditions needed in order for the program to achieve its planned success. The authors share a process of observation and focus groups that allowed them to probe some of the links of the program logic model. The result presented is an “enhanced” logic model that reveals possible moderators of program success, which leads to the development of evaluation questions that will be used in a subsequent large-scale evaluation. The value of using an enhanced logic model for program evaluation leading to program policy changes and improvement is underscored.

Download Full-text