Back to the future: Considerations in use and reporting of the retrospective pretest

Retrospective pretests ask respondents to report after an intervention on their aptitudes, knowledge, or beliefs before the intervention. A primary reason to administer a retrospective pretest is that in some situations, program participants may over the course of an intervention revise or recalibrate their prior understanding of program content, with the result that their posttest scores are lower than their traditional pretest scores, even though their understanding or abilities have increased. This phenomenon is called response-shift bias. The existence of response-shift bias is undisputed, but it does not always occur, and use of the retrospective pretest in place of a traditional pretest often introduces new problems. In this commentary, I provide a brief overview of the literature on response-shift bias and discuss common pitfalls in the use and reporting of retrospective pretest results, including a failure to consider multiple factors that may affect all test scores, as well as claims that retrospective pretests are less biased than traditional pretests, provide more accurate estimates of effects, and are necessarily superior to traditional pretests in program evaluation. I comment on the article by Little et al. (2019) in this issue in light of the literature on retrospective pretests and discuss the need for a theoretical framework to guide research on response-shift bias. The goal of the commentary is to provide readers with an informed and critical lens through which to evaluate and use retrospective pretest methods.

Testing the Retrospective Pretest with High School Youth in Out-of-School Time Programs

Practitioners and evaluators face several constraints in conducting rigorous evaluations to determine program effect. Researchers have offered the retrospective pretest/posttest design as a remedy to curb response-shift bias and better estimate program effects. This article presents an example of how After School Matters (ASM) tested the use of retrospective pretest/posttest design for evaluating out-of-school time (OST) programs for high school youth participants. Differences between traditional pretest and retrospective pretest scores were statistically significant, but effect sizes were negligible, indicating that both pretests yielded similar results. Interviews with youth led to 3 key findings that have implications for ASM using retrospective pretests with youth: response-shift bias was more prominent in youth interviews than in quantitative findings, youth recommended reordering the questions so that the retrospective pretest appears first to increase comprehension, and acquiescence bias emerged in the interviews. This study demonstrates that the retrospective pretest/posttest design can be an alternative to the traditional pretest/posttest design for OST at ASM. These findings are important for ASM and other youth-serving organizations, which often have limited capacity to survey youth multiple times within 1 program session.

Help Me Grow Utah and the impact on family protective factors development

Purpose The purpose of this paper is to examine the development of family protective factors in participants of Help Me Grow Utah (HMGU), a community-based system that promotes child development, seeks early detection of developmental delays, and links families to services. Design/methodology/approach In this paper, standard quasi-experimental survey design was utilized. HMGU and control group participants completed the FRIENDS Protective Factors Survey, which was slightly modified into a retrospective pre-test and post-test format to address previous survey concerns of response-shift bias, self-serving assessments, and family maturation. Participants were asked to respond to ten questions at present and then again from the perspective of two years previous. Findings Participants in HMGU had statistically significant increases in protective factor scores in all but one subscale, with dramatic increases in two subscale questions on knowledge of parenting and child development. Control group scores statistically increased in four subscales, albeit at lower rates than HMGU participants. Interestingly, control group scores on two subscale questions relating to child maltreatment risk were significantly lower on post-tests as compared to their retrospective pre-test scores. Research limitations/implications Participants in HMGU clearly increased in the development of protective factors. Replication of this study is recommended and the need for a control group in protective factor studies is imperative. Practical implications Findings from this study suggest that child services focused on enhancing knowledge of parenting and child development might also expect to improve protective factors. One-on-one care coordination with families seems particularly effective. The findings might also benefit other social programs as they utilize retrospective pre-test, post-test, and control groups in their evaluations. Originality/value HMGU is the first affiliate to utilize retrospective pre-test/post-test methodology, which can overcome confounding results attributable to response-shift bias. Also, the use of a control group affords inclusion of natural maturation in considering findings.

A COMPARISON OF STANDARD AND RETROSPECTIVE PRE-POST TESTING FOR MEASURING THE CHANGES IN SCIENCE TEACHING EFFICACY BELIEFS

Thirty-nine American and 78 Turkish preservice elementary teachers’ personal science teaching efficacy (PSTE) beliefs were investigated during science methods courses with standard and retrospective pre-post testing methods. Significant differences in the PSTE gain scores, which indicate the changes in the mean PSTE scores from standard/retrospective pretests to the posttest, were found between the standard and retrospective measurements in both samples. Significant differences between the standard and retrospectively measured gain scores were detected among all subgroups under study, which were formed by participants’ PSTE levels and gender. It has been concluded that the differences between the standard and retrospectively measured PSTE gain scores are due to the difference in the nature of these measurement methods and can be seen in most research samples in educational studies around the world. The findings of this study suggest that the response-shift bias should be considered as a common threat to validity for research studies measuring self-efficacy beliefs with the standard pre-post testing method. Key words: personal science teaching efficacy, preservice elementary teacher, response-shift bias, retrospective pretest.

Response Inhibition Modulates Response Conflict in Task Switching

Although response repetition (RR) effects vary considerably between conditions and studies, little is known about the causes. Recently, RR costs on task-switch trials have been found to be larger for incongruent stimuli that activate both alternative responses than for neutral ones. Here, we investigated if this modulation can be explained by an amplification of response conflict account (ARC). It assumes that a response-shift bias that is responsible for the basic RR costs amplifies the response conflict induced by incongruent stimuli specifically on trials where the response repeats. Consequently, RR costs are increased for incongruent stimuli. Because supporting evidence for this account was restricted to task-shift trials, we tested if the ARC account holds also more generally, that is, on task-repetition trials. To this end, we applied a rather common alternating-runs paradigm and presented neutral and incongruent stimuli. Results show that the congruency effect was larger on RR trials than on RS trials. Because this relation was independent of task transition, it is consistent with the idea that, in order to promote behavioral flexibility in task-switching contexts, a general response-shift bias is induced by inhibiting the previous response.