Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Background The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge. Methods Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples. Results The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice. Conclusions This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis. Highlights Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study. Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment. The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI. Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Prioritizing Research in an Era of Personalized Medicine: The Potential Value of Unexplained Heterogeneity

Background Clinical care is moving from a “one size fits all” approach to a setting in which treatment decisions are based on individual treatment response, needs, preferences, and risk. Research into personalized treatment strategies aims to discover currently unknown markers that identify individuals who would benefit from treatments that are nonoptimal at the population level. Before investing in research to identify these markers, it is important to assess whether such research has the potential to generate value. Thus, this article aims to develop a framework to prioritize research into the development of new personalized treatment strategies by creating a set of measures that assess the value of personalizing care based on a set of unknown patient characteristics. Methods Generalizing ideas from the value of heterogeneity framework, we demonstrate 3 measures that assess the value of developing personalized treatment strategies. The first measure identifies the potential value of personalizing medicine within a given disease area. The next 2 measures highlight specific research priorities and subgroup structures that would lead to improved patient outcomes from the personalization of treatment decisions. Results We graphically present the 3 measures to perform sensitivity analyses around the key drivers of value, in particular, the correlation between the individual treatment benefits across the available treatment options. We illustrate these 3 measures using a previously published decision model and discuss how they can direct research in personalized medicine. Conclusion We discuss 3 measures that form the basis of a novel framework to prioritize research into novel personalized treatment strategies. Our novel framework ensures that research targets personalized treatment strategies that have high potential to improve patient outcomes and health system efficiency. Highlights It is important to undertake research prioritization before conducting any research that aims to discover novel methods (e.g., biomarkers) for personalizing treatment. The value of unexplained heterogeneity can highlight disease areas in which personalizing treatment can be valuable and determine key priorities within that area. These priorities can be determined under assumptions of the magnitude of the individual-level treatment effect, which we explore in sensitivity analyses.

The Electronic Health Record as the Primary Data Source in a Pragmatic Trial: A Case Study

Introduction. Pragmatic clinical trials test interventions in patients representative of real-world medical practice and reduce data collection costs by using data recorded in the electronic health record (EHR) during usual care. We describe our experience using the EHR to measure the primary outcome of a pragmatic trial, hospital readmissions, and important clinical covariates. Methods. The trial enrolled patients recently discharged from the hospital for treatment of heart failure to test whether automated daily monitoring integrated into the EHR could reduce readmissions. The study team used data from the EHR and several data systems that drew on the EHR, supplemented by the hospital admissions files of three states. Results. Almost three-quarters of enrollees’ readmissions over the 12-mo trial period were captured by the EHRs of the study hospitals. State data, which took 7 mo to more than 2 y from first contact to receipt of first data, provided the remaining one-quarter. Considerable expertise was required to resolve differences between the 2 data sources. Common covariates used in trial analyses, such as weight and body mass index during the index hospital stay, were available for >97% of enrollees from the EHR. Ejection fraction, obtained from echocardiograms, was available for only 47.6% of enrollees within the 6-mo window that would likely be expected in a traditional trial. Discussion. In this trial, patient characteristics and outcomes were collected from existing EHR systems, but, as usual for EHRs, they could not be standardized for date or method of measurement and required substantial time and expertise to collect and curate. Hospital admissions, the primary trial outcome, required additional effort to locate and use supplementary sources of data. Highlights Electronic health records are not a single system but a series of overlapping and legacy systems that require time and expertise to use efficiently. Commonly measured patient characteristics such as weight and body mass index are relatively easy to locate for most trial enrollees but less common characteristics, like ejection fraction, are not. Acquiring essential supplementary data—in this trial, state data on hospital admission—can be a lengthy and difficult process.

Physicians under Pressure: Evidence from Antibiotics Prescribing in England

Background Many physicians are experiencing increasing demands from both their patients and society. Evidence is scarce on the consequences of the pressure on physicians’ decision making. We present a theoretical framework and predict that increasing pressure may make physicians disregard societal welfare when treating patients. Setting We test our prediction on general practitioners’ antibiotic-prescribing choices. Because prescribing broad-spectrum antibiotics does not require microbiological testing, it can be performed more quickly than prescribing for narrow-spectrum antibiotics and is therefore often preferred by the patient. In contrast, from a societal perspective, inappropriate prescribing of broad-spectrum antibiotics should be minimized as it may contribute to antimicrobial resistance in the general population. Methods We combine longitudinal survey data and administrative data from 2010 to 2017 to create a balanced panel of up to 1072 English general practitioners (GPs). Using a series of linear models with GP fixed effects, we estimate the importance of different sources of pressure for GPs’ prescribing. Results We find that the percentage of broad-spectrum antibiotics prescribed increases by 6.4% as pressure increases on English GPs. The link between pressure and prescribing holds for different sources of pressure. Conclusions Our findings suggest that there may be societal costs of physicians working under pressure. Policy makers need to take these costs into account when evaluating existing policies as well as when introducing new policies affecting physicians’ work pressure. An important avenue for further research is also to determine the underlying mechanisms related to the different sources of pressure.JEL-code: I11, J28, J45 Highlights Many physicians are working under increasing pressure. We test the importance of pressure on physicians’ prescribing of antibiotics. The prescribed rate of broad-spectrum antibiotics increases with pressure. Policy makers should be aware of the societal costs of pressured physicians. [Formula: see text]

An Efficient Method for Computing Expected Value of Sample Information for Survival Data from an Ongoing Trial

Background Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial. In this article, we develop new methods for computing the EVSI of extending an existing trial’s follow-up, first for an assumed survival model and then extending to capture uncertainty about the true survival model. Methods We developed a nested Markov Chain Monte Carlo procedure and a nonparametric regression-based method. We compared the methods by computing single-model and model-averaged EVSI for collecting additional follow-up data in 2 synthetic case studies. Results There was good agreement between the 2 methods. The regression-based method was fast and straightforward to implement, and scales easily included any number of candidate survival models in the model uncertainty case. The nested Monte Carlo procedure, on the other hand, was extremely computationally demanding when we included model uncertainty. Conclusions We present a straightforward regression-based method for computing the EVSI of extending an existing trial’s follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. EVSI for ongoing trials can help decision makers determine whether early patient access to a new technology can be justified on the basis of the current evidence or whether more mature evidence is needed. Highlights Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life-expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial. In this article, we have developed new methods for computing the EVSI of extending a trial’s follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. We extend a previously described nonparametric regression-based method for computing EVSI, which we demonstrate in synthetic case studies is fast, straightforward to implement, and scales easily to include any number of candidate survival models in the EVSI calculations. The EVSI methods that we present in this article can quantify the need for collecting additional follow-up data before making an adoption decision given any decision-making context.

A Revised Framework to Evaluate the Consistency Assumption Globally in a Network of Interventions

Background The unrelated mean effects (UME) model has been proposed for evaluating the consistency assumption globally in the network of interventions. However, the UME model does not accommodate multiarm trials properly and omits comparisons between nonbaseline interventions in the multiarm trials not investigated in 2-arm trials. Methods We proposed a refinement of the UME model that tackles the limitations mentioned above. We also accompanied the scatterplots on the posterior mean deviance contributions of the trial arms under the network meta-analysis (NMA) and UME models with Bland-Altman plots to detect outlying trials contributing to poor model fit. We applied the refined and original UME models to 2 networks with multiarm trials. Results The original UME model omitted more than 20% of the observed comparisons in both networks. The thorough inspection of the individual data points’ deviance contribution using complementary plots in conjunction with the measures of model fit and the estimated between-trial variance indicated that the refined and original UME models revealed possible inconsistency in both examples. Conclusions The refined UME model allows proper accommodation of the multiarm trials and visualization of all observed evidence in complex networks of interventions. Furthermore, considering several complementary plots to investigate deviance helps draw informed conclusions on the possibility of global inconsistency in the network. Highlights We have refined the unrelated mean effects (UME) model to incorporate multiarm trials properly and to estimate all observed comparisons in complex networks of interventions. Forest plots with posterior summaries of all observed comparisons under the network meta-analysis and refined UME model can uncover the consequences of potential inconsistency in the network. Using complementary plots to investigate the individual data points’ deviance contribution in conjunction with model fit measures and estimated heterogeneity aid in detecting possible inconsistency.

Perceived Social Norms Guide Health Care Decisions for Oneself and Others: A Cross-Sectional Experiment in a US Online Panel

Background: Global aging has increased the reliance on surrogates to make health care decisions for others. We investigated the differences between making health care decisions and predicting health care decisions, self-other differences for made and predicted health care decisions, and the roles of perceived social norms, emotional closeness, empathy, age, and gender. Methods: Participants ( N = 2037) from a nationally representative US panel were randomly assigned to make or to predict a health care decision. They were also randomly assigned to 1 of 5 recipients: themselves, a loved one 60 y or older, a loved one younger than 60 y, a distant acquaintance 60 y or older, or a distant acquaintance younger than 60 y. Hypothetical health care scenarios depicted choices between relatively safe lower-risk treatments with a good chance of yielding mild health improvements versus higher-risk treatments that offered a moderate chance of substantial health improvements. Participants reported their likelihood of choosing lower- versus higher-risk treatments, their perceptions of family and friends’ approval of risky health care decisions, and their empathy. Results: We present 3 key findings. First, made decisions involved less risk taking than predicted decisions, especially for distant others. Second, predicted decisions were similar for others and oneself, but made decisions were less risk taking for others than oneself. People predicted that loved ones would be less risk taking than distant others would be. Third, perceived social norms were more strongly associated than empathy with made and predicted decisions. Limitations: Hypothetical scenarios may not adequately represent emotional processes in health care decision making. Conclusions: Perceived social norms may sway people to take less risk in health care decisions, especially when making decisions for others. These findings have implications for improving surrogate decision making. Highlights People made less risky health care decisions for others than for themselves, even though they predicted others would make decisions similar to their own. This has implications for understanding how surrogates apply the substituted judgment standard when making decisions for patients. Perceived social norms were more strongly related to decisions than treatment-recipient (relationship closeness, age) and decision-maker (age, gender, empathy) characteristics. Those who perceived that avoiding health care risks was valued by their social group were less likely to choose risky medical treatments. Understanding the power of perceived social norms in shaping surrogates’ decisions may help physicians to engage surrogates in shared decision making. Knowledge of perceived social norms may facilitate the design of decision aids for surrogates.

Exploring Structural Uncertainty and Impact of Health State Utility Values on Lifetime Outcomes in Diabetes Economic Simulation Models: Findings from the Ninth Mount Hood Diabetes Quality-of-Life Challenge

Background Structural uncertainty can affect model-based economic simulation estimates and study conclusions. Unfortunately, unlike parameter uncertainty, relatively little is known about its magnitude of impact on life-years (LYs) and quality-adjusted life-years (QALYs) in modeling of diabetes. We leveraged the Mount Hood Diabetes Challenge Network, a biennial conference attended by international diabetes modeling groups, to assess structural uncertainty in simulating QALYs in type 2 diabetes simulation models. Methods Eleven type 2 diabetes simulation modeling groups participated in the 9th Mount Hood Diabetes Challenge. Modeling groups simulated 5 diabetes-related intervention profiles using predefined baseline characteristics and a standard utility value set for diabetes-related complications. LYs and QALYs were reported. Simulations were repeated using lower and upper limits of the 95% confidence intervals of utility inputs. Changes in LYs and QALYs from tested interventions were compared across models. Additional analyses were conducted postchallenge to investigate drivers of cross-model differences. Results Substantial cross-model variability in incremental LYs and QALYs was observed, particularly for HbA1c and body mass index (BMI) intervention profiles. For a 0.5%-point permanent HbA1c reduction, LY gains ranged from 0.050 to 0.750. For a 1-unit permanent BMI reduction, incremental QALYs varied from a small decrease in QALYs (−0.024) to an increase of 0.203. Changes in utility values of health states had a much smaller impact (to the hundredth of a decimal place) on incremental QALYs. Microsimulation models were found to generate a mean of 3.41 more LYs than cohort simulation models ( P = 0.049). Conclusions Variations in utility values contribute to a lesser extent than uncertainty captured as structural uncertainty. These findings reinforce the importance of assessing structural uncertainty thoroughly because the choice of model (or models) can influence study results, which can serve as evidence for resource allocation decisions. Highlights The findings indicate substantial cross-model variability in QALY predictions for a standardized set of simulation scenarios and is considerably larger than within model variability to alternative health state utility values (e.g., lower and upper limits of the 95% confidence intervals of utility inputs). There is a need to understand and assess structural uncertainty, as the choice of model to inform resource allocation decisions can matter more than the choice of health state utility values.

Can Machine Learning from Real-World Data Support Drug Treatment Decisions? A Prediction Modeling Case for Direct Oral Anticoagulants

Background: Decision making for the “best” treatment is particularly challenging in situations in which individual patient response to drugs can largely differ from average treatment effects. By estimating individual treatment effects (ITEs), we aimed to demonstrate how strokes, major bleeding events, and a composite of both could be reduced by model-assisted recommendations for a particular direct oral anticoagulant (DOAC). Methods: In German claims data for the calendar years 2014–2018, we selected 29 901 new users of the DOACs rivaroxaban and apixaban. Random forests considered binary events within 1 y to estimate ITEs under each DOAC according to the X-learner algorithm with 29 potential effect modifiers; treatment recommendations were based on these estimated ITEs. Model performance was evaluated by the c-for-benefit statistics, absolute risk reduction (ARR), and absolute risk difference (ARD) by trial emulation. Results: A significant proportion of patients would be recommended a different treatment option than they actually received. The stroke model significantly discriminated patients for higher benefit and thus indicated improved decisions by reduced outcomes (c-for-benefit: 0.56; 95% confidence interval [0.52; 0.60]). In the group with apixaban recommendation, the model also improved the composite endpoint (ARR: 1.69 % [0.39; 2.97]). In trial emulations, model-assisted recommendations significantly reduced the composite event rate (ARD: −0.78 % [−1.40; −0.03]). Conclusions: If prescribers are undecided about the potential benefits of different treatment options, ITEs can support decision making, especially if evidence is inconclusive, risk-benefit profiles of therapeutic alternatives differ significantly, and the patients’ complexity deviates from “typical” study populations. In the exemplary case for DOACs and potentially in other situations, the significant impact could also become practically relevant if recommendations were available in an automated way as part of decision making. Highlights It was possible to calculate individual treatment effects (ITEs) from routine claims data for rivaroxaban and apixaban, and the characteristics between the groups with recommendation for one or the other option differed significantly. ITEs resulted in recommendations that were significantly superior to usual (observed) treatment allocations in terms of absolute risk reduction, both separately for stroke and in the composite endpoint of stroke and major bleeding. When similar patients from routine data were selected (precision cohorts) for patients with a strong recommendation for one option or the other, those similar patients under the respective recommendation showed a significantly better prognosis compared with the alternative option. Many steps may still be needed on the way to clinical practice, but the principle of decision support developed from routine data may point the way toward future decision-making processes.

Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials

Introduction Adaptive designs allow changes to an ongoing trial based on prespecified early examinations of accrued data. Opportunities are potentially being missed to incorporate health economic considerations into the design of these studies. Methods We describe how to estimate the expected value of sample information for group sequential design adaptive trials. We operationalize this approach in a hypothetical case study using data from a pilot trial. We report the expected value of sample information and expected net benefit of sampling results for 5 design options for the future full-scale trial including the fixed-sample-size design and the group sequential design using either the Pocock stopping rule or the O’Brien-Fleming stopping rule with 2 or 5 analyses. We considered 2 scenarios relating to 1) using the cost-effectiveness model with a traditional approach to the health economic analysis and 2) adjusting the cost-effectiveness analysis to incorporate the bias-adjusted maximum likelihood estimates of trial outcomes to account for the bias that can be generated in adaptive trials. Results The case study demonstrated that the methods developed could be successfully applied in practice. The results showed that the O’Brien-Fleming stopping rule with 2 analyses was the most efficient design with the highest expected net benefit of sampling in the case study. Conclusions Cost-effectiveness considerations are unavoidable in budget-constrained, publicly funded health care systems, and adaptive designs can provide an alternative to costly fixed-sample-size designs. We recommend that when planning a clinical trial, expected value of sample information methods be used to compare possible adaptive and nonadaptive trial designs, with appropriate adjustment, to help justify the choice of design characteristics and ensure the cost-effective use of research funding. Highlights Opportunities are potentially being missed to incorporate health economic considerations into the design of adaptive clinical trials. Existing expected value of sample information analysis methods can be extended to compare possible group sequential and nonadaptive trial designs when planning a clinical trial. We recommend that adjusted analyses be presented to control for the potential impact of the adaptive designs and to maintain the accuracy of the calculations. This approach can help to justify the choice of design characteristics and ensure the cost-effective use of limited research funding.