Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Background The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge. Methods Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples. Results The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice. Conclusions This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis. Highlights Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study. Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment. The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI. Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Interventions Within the Scope of Occupational Therapy to Address Preventable Adverse Events in Inpatient and Home Health Postacute Care Settings: A Systematic Review

Importance: Practitioners need to be familiar with, and involved in, managing quality-related adverse events in postacute care. Objective: To determine interventions within the scope of occupational therapy that address preventable adverse events in adult postacute inpatient and home health settings. Data Sources: Articles published from January 1995 through 2019 identified through searches of MEDLINE, PsycINFO, CINAHL, OTseeker, and Cochrane databases. Study Selection and Data Collection: Articles were collected, evaluated, and analyzed by two independent reviewers. They were assessed and synthesized with a goal of informing clinical practice. Findings: Twenty-four articles were included in the review. Of the 10 Centers for Medicare & Medicaid Services preventable adverse events, 6 were addressed: diabetes management (n = 2), dysphagia (n = 5), infection control (n = 1), pressure ulcers (n = 6), falls (n = 5), and discharge management (n = 5). There was strong strength of evidence that exercise programs should, when appropriate, be implemented in both inpatient and home health settings to decrease the risk of falls. There was moderate strength of evidence that practitioners could consider implementing a facility wide evidence-based pressure ulcer program; providing multidisciplinary rehabilitation and swallow strengthening exercises for dysphagia; implementing a multidisciplinary, multicomponent falls program; and using a manualized depression intervention in home health to decrease hospital readmission. Conclusions and Relevance: The review highlights the importance of preventable adverse events and of occupational therapy practitioners acknowledging and managing these events to enhance health outcomes and to control health care costs. What This Article Adds: Many interventions typically performed by occupational therapy practitioners address preventable adverse events. The review highlights the importance of practitioners being aware of this category of impairment or injury.

Occupational Therapy and the IMPACT Act: Part 2. A Systematic Review of Evidence for Functional Status, Medication Reconciliation, and Skin Integrity Interventions

Importance: Interventions that promote function, medication reconciliation, and skin integrity assist occupational therapy practitioners in demonstrating professional value, improving quality, and reducing health care costs. Objective: In this systematic review, we focus on three outcome areas of the Improving Medicare Post-Acute Care Transformation (IMPACT) Act of 2014: functional status, medication reconciliation, and skin integrity. Data Sources: We conducted a search of the literature published between 2009 and 2019 in CINAHL, Cochrane, MEDLINE, PsycINFO, OTseeker, and Scopus. We also hand searched the systematic reviews and meta-analyses in our search results for articles that met our inclusion criteria. Study Selection and Data Collection: This study used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Findings: We found 47 articles that address the three outcome areas. Regarding functional status, low strength of evidence is available for cognition and functional mobility interventions to support functional performance, moderate strength of evidence supports interventions for vision, and moderate evidence supports task-oriented and individualized interventions to promote activities of daily living (ADL) outcomes among people with neurological conditions. Strong strength of evidence supports individualized occupational therapy interventions focusing on medication adherence. Low strength of evidence was found for occupational therapy interventions to reduce pressure ulcers and promote skin integrity. Conclusion and Relevance: The evidence supports occupational therapy interventions to improve functional status in ADLs and medication management. Additional research is needed that examines the outcomes of occupational therapy interventions for other areas of function and skin integrity. What This Article Adds: We found evidence to support occupational therapy interventions that align with value-based measures in the three outcome areas of interest. The effectiveness of these interventions highlights the viability of occupational therapy as an essential profession and the worth of occupational therapy to the public, potential clients, and payers.

Occupational Therapy and the IMPACT Act: Part 1. A Systematic Review of Evidence for Fall Prevention and Reduction, Community Discharge and Reintegration, and Readmission Prevention Interventions

Importance: Interventions that prevent falls, facilitate discharge after hospitalization, and reduce hospital readmissions assist occupational therapy practitioners in demonstrating professional value, improving quality, and reducing costs. Objective: In this systematic review, we address three outcome areas of the Improving Medicare Post-Acute Care Transformation (IMPACT) Act of 2014: prevention and reduction of falls, facilitation of community discharge and reintegration, and prevention of hospital readmission. Data Sources: We conducted a search of the literature published between 2009 and 2019. Study Selection and Data Collection: We developed operational definitions to help us identify articles that answered the search question for each outcome area. This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Findings: We found 53 articles that address the three outcome areas. Regarding the prevention and reduction of falls, low strength of evidence is available for interventions focusing on a single fall risk and for customized interventions addressing multiple risks. Moderate strength of evidence supports structured community fall risk prevention interventions. Low strength of evidence was found for community discharge and reintegration interventions that include physical activity and educational programming. Low to moderate strength of evidence was found for readmission prevention interventions for patients with four types of condition. Conclusion and Relevance: Several intervention themes in the three outcome areas of interest are supported by few studies or by studies with a moderate risk of bias. Additional research is needed that supports the value of occupational therapy interventions in these outcome areas. What This Article Adds: Our study provides important insights into the state of the evidence related to occupational therapy interventions to address three outcome areas of the IMPACT Act.

Occupational Therapy and the IMPACT Act: Part 1. A Systematic Review of Evidence for Fall Prevention and Reduction, Community Discharge and Reintegration, and Readmission Prevention Interventions

Importance: Interventions that prevent falls, facilitate discharge after hospitalization, and reduce hospital readmissions assist occupational therapy practitioners in demonstrating professional value, improving quality, and reducing costs. Objective: In this systematic review, we address three outcome areas of the Improving Medicare Post-Acute Care Transformation (IMPACT) Act of 2014: prevention and reduction of falls, facilitation of community discharge and reintegration, and prevention of hospital readmission. Data Sources: We conducted a search of the literature published between 2009 and 2019. Study Selection and Data Collection: We developed operational definitions to help us identify articles that answered the search question for each outcome area. This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Findings: We found 53 articles that address the three outcome areas. Regarding the prevention and reduction of falls, low strength of evidence is available for interventions focusing on a single fall risk and for customized interventions addressing multiple risks. Moderate strength of evidence supports structured community fall risk prevention interventions. Low strength of evidence was found for community discharge and reintegration interventions that include physical activity and educational programming. Low to moderate strength of evidence was found for readmission prevention interventions for patients with four types of condition. Conclusion and Relevance: Several intervention themes in the three outcome areas of interest are supported by few studies or by studies with a moderate risk of bias. Additional research is needed that supports the value of occupational therapy interventions in these outcome areas. What This Article Adds: Our study provides important insights into the state of the evidence related to occupational therapy interventions to address three outcome areas of the IMPACT Act.

The Number of Unanimous Witnesses who Identify a Suspect from a Lineup Influences Mock Jurors

<p>During a criminal proceeding, jurors need to weigh up the presented evidence and determine a verdict. Research has shown that witness identification evidence is compelling to jurors, despite the fact that it can be unreliable. How reliable are the combined lineup decisions gathered from multiple witnesses? Generally, the more witnesses who identify the same person from a lineup, the more likely that person is guilty. But recent theoretical evidence suggests that a greater number of witnesses identifying the same person from a biased lineup can indicate that person is actually less likely to be guilty than if there were a smaller number of witnesses identifying that person (Gunn et al., 2016). As the number of agreeing witnesses increases, the more likely that agreement is caused by the lineup bias, rather than consistent witness memories of the crime. In this thesis, I examined how unanimity and lineup bias influenced jurors’ perceptions of guilt. Subjects who saw a biased lineup gave lower ratings of guilt compared to subjects that were shown a lineup that had no obvious bias. In addition, warning subjects that a lineup was biased led them to give lower guilt ratings than subjects who did not receive a warning. Subjects who were told there were two witnesses who identified the police suspect gave higher guilt ratings than subjects who were told there was one witness who identified the police suspect, but only when the lineup was clearly not biased. Subjects’ guilt ratings were not significantly greater in conditions with more than two unanimous witnesses identifying the police suspect. It seems subjects had a limit of certainty based on changes in witness numbers alone. We also found that the way in which witness numbers were presented to subjects influenced guilt ratings. When we presented witnesses coming forward in different groups and on different days, subjects shifted their guilt ratings upwards. When the number of witnesses decreased during the experiment, subjects did not decrease their guilt ratings to the same extent as those subjects in conditions in which the number of witnesses increased by the same magnitude. This finding is consistent with the literature on confirmation bias and the story model of juror decision-making—subjects likely formed an initial belief that the identified suspect was guilty and subsequent evidence was evaluated against that belief (Nickerson, 1988; Pennington & Hastie, 1993). The finding that presenting witnesses coming forward in separate groups increased subjects’ guilt ratings adds to the literature showing that jurors are influenced by irrelevant information presented to them during a proceeding. This research also demonstrates that future research should examine strength of evidence manipulations over multiple levels—rather than as dichotomous “strong” and “weak” extremes.</p>

The Number of Unanimous Witnesses who Identify a Suspect from a Lineup Influences Mock Jurors

<p>During a criminal proceeding, jurors need to weigh up the presented evidence and determine a verdict. Research has shown that witness identification evidence is compelling to jurors, despite the fact that it can be unreliable. How reliable are the combined lineup decisions gathered from multiple witnesses? Generally, the more witnesses who identify the same person from a lineup, the more likely that person is guilty. But recent theoretical evidence suggests that a greater number of witnesses identifying the same person from a biased lineup can indicate that person is actually less likely to be guilty than if there were a smaller number of witnesses identifying that person (Gunn et al., 2016). As the number of agreeing witnesses increases, the more likely that agreement is caused by the lineup bias, rather than consistent witness memories of the crime. In this thesis, I examined how unanimity and lineup bias influenced jurors’ perceptions of guilt. Subjects who saw a biased lineup gave lower ratings of guilt compared to subjects that were shown a lineup that had no obvious bias. In addition, warning subjects that a lineup was biased led them to give lower guilt ratings than subjects who did not receive a warning. Subjects who were told there were two witnesses who identified the police suspect gave higher guilt ratings than subjects who were told there was one witness who identified the police suspect, but only when the lineup was clearly not biased. Subjects’ guilt ratings were not significantly greater in conditions with more than two unanimous witnesses identifying the police suspect. It seems subjects had a limit of certainty based on changes in witness numbers alone. We also found that the way in which witness numbers were presented to subjects influenced guilt ratings. When we presented witnesses coming forward in different groups and on different days, subjects shifted their guilt ratings upwards. When the number of witnesses decreased during the experiment, subjects did not decrease their guilt ratings to the same extent as those subjects in conditions in which the number of witnesses increased by the same magnitude. This finding is consistent with the literature on confirmation bias and the story model of juror decision-making—subjects likely formed an initial belief that the identified suspect was guilty and subsequent evidence was evaluated against that belief (Nickerson, 1988; Pennington & Hastie, 1993). The finding that presenting witnesses coming forward in separate groups increased subjects’ guilt ratings adds to the literature showing that jurors are influenced by irrelevant information presented to them during a proceeding. This research also demonstrates that future research should examine strength of evidence manipulations over multiple levels—rather than as dichotomous “strong” and “weak” extremes.</p>

Assessing the Global and Local Uncertainty of Scientific Evidence in the Presence of Model Misspecification

Scientists need to compare the support for models based on observed phenomena. The main goal of the evidential paradigm is to quantify the strength of evidence in the data for a reference model relative to an alternative model. This is done via an evidence function, such as ΔSIC, an estimator of the sample size scaled difference of divergences between the generating mechanism and the competing models. To use evidence, either for decision making or as a guide to the accumulation of knowledge, an understanding of the uncertainty in the evidence is needed. This uncertainty is well characterized by the standard statistical theory of estimation. Unfortunately, the standard theory breaks down if the models are misspecified, as is commonly the case in scientific studies. We develop non-parametric bootstrap methodologies for estimating the sampling distribution of the evidence estimator under model misspecification. This sampling distribution allows us to determine how secure we are in our evidential statement. We characterize this uncertainty in the strength of evidence with two different types of confidence intervals, which we term “global” and “local.” We discuss how evidence uncertainty can be used to improve scientific inference and illustrate this with a reanalysis of the model identification problem in a prominent landscape ecology study using structural equations.