The pandemic silver lining: preparing osteopathic learners to address healthcare needs using telehealth

Context During the COVID-19 pandemic, many clinicians quickly adapted their way of practicing patient care by offering telehealth and virtual office visits while simultaneously having to minimize direct patient care. The shift in direct clinical learning opportunities provided to third- and fourth-year medical students required a shift in the educational curriculum to develop learner skills around the appropriate use of telehealth in patient care. Objectives The aim of this project was to provide exposure to students so they could learn the telemedicine equipment and best practices, and how to identify infectious diseases to improve access to care and meet the needs of the patient. Methods In July and August of 2020, the Indiana Area Health Education Centers Program partnered with Marian University College of Osteopathic Medicine (MUCOM) to support a 1 day telehealth simulation (online curriculum, group lecture, and two standardized patient encounters) into their clerkship curriculum. We utilized a retrospective pretest-posttest to assess changes in learner knowledge around telehealth after the program. At the conclusion of the telehealth training program, students were asked to complete a retrospective pretest-posttest assessing their level of preparedness to utilize telehealth equipment, their preparedness to demonstrate “telehealth best practices” in a manner consistent with protecting patient (and data) privacy, their confidence to utilize telehealth for identification of infectious diseases, and their confidence to utilize telehealth to identify proper treatment plans. Results A total of 96 learners completed the program in 2020. Posttest results demonstrate a statistically significant (p<0.05) improvement for learners’ self-reported level of preparedness to utilize telehealth equipment, their preparedness to demonstrate “telehealth best practices” in a manner consistent with protecting patient (and data) privacy, their confidence to utilize telehealth for identification of infectious diseases, and their confidence to utilize telehealth to identify proper treatment plans. Conclusions Our telehealth curriculum involving a video, interactive learning session, and two standardized patient experiences provided osteopathic medical learners with realistic simulated case scenarios to work through in effort to improve their knowledge and self-efficacy around the utilization of telehealth in practice.

Anxiety and Social Support as Predictors of Student Academic Motivation During the COVID-19

In this study we examined whether parents’ perceptions of students’ anxiety as well as perceived support from both teachers and classmates were predictive of changes in students’ academic motivation during the first wave of COVID-19. To this end, we used a retrospective pretest-posttest design together with a latent change score model to analyze our data. From April to May of 2020, 394 Portuguese parents of students in grades 1–9 participated in this study. Our results showed that students’ anxiety and teachers’ social support, as perceived by parents, were highly significant predictors of academic motivation changes. Specifically, we found a negative effect of anxiety and a positive effect of teachers’ social support on students’ academic motivation. Our results did not show, however, a significant predictive role of classmates’ social support. This study provides an important contribution to further understand the intrapersonal and interpersonal factors that are associated with the decline of students’ academic motivation during the COVID-19 pandemic. The pivotal role of teachers in sustaining students’ academic motivation and other relevant educational implications for the ongoing pandemic are discussed.

Including Diverse Stakeholder Voices in Youth Character Program Evaluation

Although experts agree that diverse stakeholder inclusion enhances quality and equity in evaluation designs and implementation, diverse voices are often omitted. Particularly antithetical to principles of youth character development, evaluations for these programs should strive to include voices from various social, economic, community, and demographic perspectives. One innovative national evaluation capacity building initiative, the Partnerships for Advancing Character Program Evaluation (PACE) project, paired practitioners from youth programs in community-based organizations with evaluation professionals to enhance stakeholders’ roles in evaluation. PACE promoted stakeholder identification and inclusion through group exercises, partnership work, and coaching sessions. Using a mixed methods design with interviews, retrospective pretest–posttest surveys, and observational data, triangulated data addressed diverse stakeholders in the evaluation process, diverse perspectives on program performance, and connecting diverse input to evaluation design. Postprogram findings indicate that participants included more varied and diverse stakeholder perspectives in all the three areas. Implications for programs and evaluations are discussed.

Responding to Patient-Initiated Verbal Sexual Harassment: Outcomes of a Pilot Training for Ophthalmologists

Background Sexual harassment of physicians by patients is highly prevalent and rarely reported. Little is known on how to prepare physicians to handle verbal sexual harassment that detracts from their ability to provide care but does not meet the threshold for reporting. Purpose To assess the impact of a sexual harassment workshop and toolkit for ophthalmologists and ophthalmology trainees on responding to patient-initiated verbal sexual harassment. Methods A survey study of ophthalmology faculty, fellows, and residents who participated in workshops on responding to patient-initiated verbal sexual harassment was performed at an academic center. A toolkit of strategies for response was distributed. Volunteer participants completed a retrospective pretest–posttest evaluation at the conclusion of the workshop and follow-up survey 3 weeks after the workshops on whether they experienced harassment and intervened. The pretest–posttest surveys assessed the workshop's effect on ophthalmologists' perceptions of and preparedness to respond to sexual harassment in the moment using a 5-point Likert scale, including bystander intervention. Participants described their responses observing and/or experiencing patient-initiated sexual harassment in the 3 weeks following the workshop and whether they had intervened toward the harassment. Results Ophthalmologists (n = 31) felt significantly more prepared to respond to patient-initiated sexual harassment directed toward themselves or a trainee in the moment after participating in the workshop (4.5 ± 1.63) than before (3.0 ± 1.3) with a mean change of 1.6 (95% confidence interval: 0.98–2.2, p < 0.001). After the workshop, 86.3% of participants felt mostly or completely prepared to respond to comments about their age, gender, marital status, appearance, attractiveness, a specific body part, and sexual or inappropriate jokes. Most participants (83.9%) said that they had not previously received training on techniques for responding to patient-initiated sexual harassment. Two-thirds (66.7%) of participants who experienced (n = 8) or observed (n = 13) harassment (n = 15) following the workshop intervened. All participants who intervened toward patient-initiated harassment behavior after the workshop (n = 10) found the Sexual Harassment Toolkit helpful in addressing harassment in the moment. Conclusion Participation in a brief skills-based workshop significantly improved ophthalmologists' preparedness to respond to verbal sexual harassment by patients.

The retrospective pretest–posttest design redux: On its validity as an alternative to traditional pretest–posttest measurement

We revisit the merits of the retrospective pretest–posttest (RPP) design for repeated-measures research. The underutilized RPP method asks respondents to rate survey items twice during the same posttest measurement occasion from two specific frames of reference: “now” and “then.” Individuals first report their current attitudes or beliefs following a given intervention, and next they are prompted to think back to a specific time prior to the given intervention and rate the item again retrospectively. The design addresses many of the validity concerns that plague the traditional pretest–posttest design. Particularly when measuring noncognitive constructs, the RPP design allows participants to gauge the degree of change that they experience with greater awareness and precision than a traditional approach. We review the undesirable features of traditional designs and highlight the benefits of the retrospective approach. We offer examples from two recent, original studies and conclude with the recommendation that the RPP design be employed more broadly. We also conclude with a discussion of important directions for future examination of this design.

Back to the future: Considerations in use and reporting of the retrospective pretest

Retrospective pretests ask respondents to report after an intervention on their aptitudes, knowledge, or beliefs before the intervention. A primary reason to administer a retrospective pretest is that in some situations, program participants may over the course of an intervention revise or recalibrate their prior understanding of program content, with the result that their posttest scores are lower than their traditional pretest scores, even though their understanding or abilities have increased. This phenomenon is called response-shift bias. The existence of response-shift bias is undisputed, but it does not always occur, and use of the retrospective pretest in place of a traditional pretest often introduces new problems. In this commentary, I provide a brief overview of the literature on response-shift bias and discuss common pitfalls in the use and reporting of retrospective pretest results, including a failure to consider multiple factors that may affect all test scores, as well as claims that retrospective pretests are less biased than traditional pretests, provide more accurate estimates of effects, and are necessarily superior to traditional pretests in program evaluation. I comment on the article by Little et al. (2019) in this issue in light of the literature on retrospective pretests and discuss the need for a theoretical framework to guide research on response-shift bias. The goal of the commentary is to provide readers with an informed and critical lens through which to evaluate and use retrospective pretest methods.

Testing the Retrospective Pretest with High School Youth in Out-of-School Time Programs

Practitioners and evaluators face several constraints in conducting rigorous evaluations to determine program effect. Researchers have offered the retrospective pretest/posttest design as a remedy to curb response-shift bias and better estimate program effects. This article presents an example of how After School Matters (ASM) tested the use of retrospective pretest/posttest design for evaluating out-of-school time (OST) programs for high school youth participants. Differences between traditional pretest and retrospective pretest scores were statistically significant, but effect sizes were negligible, indicating that both pretests yielded similar results. Interviews with youth led to 3 key findings that have implications for ASM using retrospective pretests with youth: response-shift bias was more prominent in youth interviews than in quantitative findings, youth recommended reordering the questions so that the retrospective pretest appears first to increase comprehension, and acquiescence bias emerged in the interviews. This study demonstrates that the retrospective pretest/posttest design can be an alternative to the traditional pretest/posttest design for OST at ASM. These findings are important for ASM and other youth-serving organizations, which often have limited capacity to survey youth multiple times within 1 program session.

Innovations for Evaluation Research: Multiform Protocols, Visual Analog Scaling, and the Retrospective Pretest–Posttest Design

In this article, we review three innovative methods: multiform protocols, visual analog scaling, and the retrospective pretest–posttest design that can be used in evaluation research. These three techniques have been proposed for decades, but unfortunately, they are still not utilized readily in evaluation research. Our goal is to familiarize researchers with these underutilized research techniques that could reduce personnel effort and costs for data collection while producing better inferences for a study. We begin by discussing their applications and special unique features. We then discuss each technique’s strengths and limitations and offer practical tips on how to better implement these methods in evaluation research. We then showcase two recent empirical studies that implement these methods in real-world evaluation research applications.