OP208 Did Health Technology Assessments Make the Wrong Call? Quantitative Bias Analysis: Alectinib versus Ceritinib in Non-Small Cell Lung Cancer

IntroductionThe German health technology assessment (HTA) rejected additional benefit of alectinib for second line (2L) ALK+ NSCLC, citing possible biases from missing ECOG performance status data and unmeasured confounding in real-world evidence (RWE) for 2L ceritinib that was submitted as a comparator to the single arm alectinib trial. Alectinib was approved in the US and therefore US post-launch RWE can be used to evaluate this HTA decision.MethodsWe compared the real-world effectiveness of alectinib with ceritinib in 2L post-crizotinib ALK+ NSCLC using the nationwide Flatiron Health electronic health record (EHR)-derived de-identified database. Using quantitative bias analysis (QBA), we estimated the strength of (i) unmeasured confounding and (ii) deviation from missing-at-random (MAR) assumptions needed to nullify any overall survival (OS) benefit.ResultsAlectinib had significantly longer median OS than ceritinib in complete case analysis. The estimated effect size (Hazard Ratio: 0.55) was robust to risk ratios of unmeasured confounder-outcome and confounder-exposure associations of <2.4.Based on tipping point analysis, missing baseline ECOG performance status for ceritinib-treated patients (49% missing) would need to be more than 3.4-times worse than expected under MAR to nullify the OS benefit observed for alectinib.ConclusionsOnly implausible levels of bias reversed our conclusions. These methods could provide a framework to explore uncertainty and aid decision-making for HTAs to enable patient access to innovative therapies.

Assess the Application of the E-Value in the Unmeasured Confounder Evaluation of Observational Pharmaceutical Studies

Public health is very important in big cities, and data analysis on public health studies is always a demanding issue that determines the study effectiveness. E-value was proposed as a standard sensitivity analysis tool to assess unmeasured confounders in observational studies, but its value is doubted. To evaluate the usefulness of E-value, in this paper, we collected 368 observational studies on drug effectiveness evaluation published from 1998 to September 2019 (out of 3426 searched studies) and evaluated the features of E-value. We selected the effects of primary outcomes or the largest effects in terms of hazard ratio, risk ratio, or odds ratio. Effects were transformed into estimated effect sizes following a standard E-value computation. In all 368 studies, the disease with the highest percentage was infections and infestations, at 21.7% (80/368). Our results showed that the median relative effect size was 1.89 (Q1-Q3: 1.41–2.95), and the corresponding median E-value was 3.19 with 95% confidence interval lower bound 1.77. Smaller studies yielded larger E-values for the effect size estimate and the relationship was considerably attenuated when considering the E-value for the lower bound of 95% confidence interval on the effect size. Notably, E-values have a monotonic, almost linear relationship with effect estimates. We found that E-value may cause misimpressions on the unmeasured confounder, and the same E-value does not reflect the varying nature of the unmeasured confounders in different studies, and there lacks a guidance on how E-value can be deemed as small or large, all of which limits the capability of E-value as a standard sensitivity analysis tool in real applications.

1273Confounding in the association between pregnancy complications and subsequent preterm birth

Background Underlying causal mechanisms that persist from pregnancy to pregnancy have potential to explain recurrence of pregnancy complications. We aimed to estimate the degree of confounding necessary to explain these associations. Methods This was a retrospective cohort study of women (n = 124,936) giving birth to their first and second singleton children in Western Australia 1998-2015, identified from the Midwives’ Notifications System. The investigated pregnancy complications were preterm birth (&lt;37 gestational weeks), pre-eclampsia, placental abruption, small-for-gestational-age and perinatal death. Adjusted relative risks (RR) and confidence intervals (CI) were reported. We simulated maternal obesity and derived E-values, a method to determine the magnitude of unmeasured confounding. Results Complications in first pregnancy were associated with higher risk of preterm birth in second pregnancy. RR’s were significantly higher when the complication was recurrent. For the association between pre-eclampsia at first term birth and subsequent preterm birth, the RR increased from 1.2 (95% CI 1.05-1.41) to 11.9 (95% CI 9.52-11.49) when the complication reoccurred. E-values were 1.73 and 23.22 respectfully. Relative risks did not change after adjustment for maternal obesity. Conclusions The strong associations between pregnancy complications and preterm birth support the assumption of shared underlying causes that persist from pregnancy to pregnancy. High E-values suggest that recurrent confounding is unlikely, as any such unmeasured confounder would have to be uncharacteristically large. Key messages Well-established single confounders cannot explain away the strong associations between complications in first pregnancy and a subsequent preterm birth.

On the use of the E-value for sensitivity analysis in epidemiologic studies

This study illustrates the use of a recently developed sensitivity index, the E-value, helpful in strengthening causal inferences in observational epidemiological studies. The E-value aims to determine the minimum required strength of association between an unmeasured confounder and an exposure/outcome to explain the observed association as non-causal. Such parameter is defined as E - v a l u e = R R + R R R R - 1, where RR is the risk ratio between the exposure and the outcome. Our work illustrates the E-value using observational data from a recently published study on the relationship between indicators of prenatal care adequacy and the outcome low birthweight. The E-value ranged between 1.45 and 5.63 according to the category and prenatal care index evaluated, showing the highest value for the “no prenatal care” category of the GINDEX index and the minimum value for “intermediate prenatal care” of the APNCU index. For “inappropriate prenatal care” (all indexes), the E-value ranged between 2.76 (GINDEX) and 4.99 (APNCU). These findings indicate that only strong confounder/low birthweight associations (more than 400% increased risk) would be able to fully explain the prenatal care vs. low birthweight association observed. The E-value is a useful, intuitive sensitivity analysis tool that may help strengthening causal inferences in epidemiological observational studies.

Performance of Logistic Regression, Propensity Scores, and Instrumental Variable for Estimating Their True Target Odds Ratios in the Absence and Presence of Unmeasured Confounder

Background: A lot of studies have compared the ability of statistical methods to control for confounding. However, a majority of studies mistakenly assumed these methods estimate the same effect. The aim of this study was to use Monte Carlo simulations to compare logistic regression, propensity scores and instrumental variable analysis for estimating their true target odds ratios in terms of bias and precision in the absence and presence of unmeasured confounder. Methods: We established the formula allowing us to compute the true odds ratio of each method. We varied the instrument’s strength and the unmeasured confounder to cover a large range of scenarios in the simulation study. We then use logistic regression, propensity score matching, propensity score adjustment and two-stage residual inclusion to obtain estimated odds ratios in each scenario. Results: In the absence of unmeasured confounder, instrumental variable without direct effect on the outcome could produce unbiased estimates as propensity score did, but the mean square errors of instrumental variable were greater. When unmeasured confounder existed, no other method could produce unbiased estimation except instrumental variable, provided that the proposed instrument is not directly related to the outcome. Using the defined instrument, which affected the outcome directly, resulted in positive biased estimation of the treatment effect and this bias was greater compared to that from other methods. Conclusions: Overall, with good implementation, instrumental variable can lead to unbiased results. However, the bias caused by violating the required assumptions of instrumental variable can overweigh the positive effect of its ability to control for unmeasured confounder.

Bias Reduction through Analysis of Competing Events (BRACE): A Novel Method to Mitigate Bias from Residual Confounding in Observational Data

ABSTRACTPurposeTo introduce a method to mitigate bias from residual confounding in non-randomized data and examine its performance under varying conditions using simulated data.MethodsWe developed a method called Bias Reduction through Analysis of Competing Events (BRACE) based on a proportional relative hazards model. We followed recommended guidelines (ADEMP) established for the conduct of simulation studies. The primary estimand of interest was the treatment effect on the composite hazard for a primary or competing event. We compared the BRACE method to a standard Cox proportional hazards regression model in the presence of an unmeasured confounder, using a parametric (Weibull) simulation model. We examined estimator distributions, bias, mean squared error (MSE), and coverage probability for both methods using ridge, box-and-whisker, forest, and zip plots, respectively. Comparisons with a hypothetical validation estimate treating the confounder as measurable were also performed.ResultsWe presented 16 simulation scenarios under varying parameters. In simulations where residual confounding was present, the BRACE method uniformly reduced both bias and MSE compared to standard Cox models. In the scenario of moderate bias with an effective but non-toxic treatment, MSE was 3.51×10−2 with the standard model vs. 0.259×10−2 with the BRACE method. In the absence of bias, the BRACE method introduced bias toward the null (2.90 x10−2) compared to the standard method (0.331×10−2), albeit with lower MSE (0.341 x10−2 vs. 0.484 x10−2, respectively). Relative to the standard approach, the BRACE method markedly improved coverage probability, but with a tendency toward overcorrection in the case of the effective but non-toxic treatment. Conclusions were similar under different parameter assumptions.ConclusionThe BRACE method can reduce bias and MSE in the setting of residual confounding.

Unsafe Injection Is Associated with Higher HIV Testing after Bayesian Adjustment for Unmeasured Confounding

Background: To apply a novel method to adjust for HIV knowledge as an unmeasured confounder for the effect of unsafe injection on future HIV testing. Methods: The data were collected from 601 HIV-negative persons who inject drugs (PWID) from a cohort in San Francisco. The panel-data generalized estimating equations (GEE) technique was used to estimate the adjusted risk ratio (RR) for the effect of unsafe injection on not being tested (NBT) for HIV. Expert opinion quantified the bias parameters to adjust for insufficient knowledge about HIV transmission as an unmeasured confounder using Bayesian bias analysis. Results: Expert opinion estimated that 2.5%–40.0% of PWID with unsafe injection had insufficient HIV knowledge; whereas 1.0%–20.0% who practiced safe injection had insufficient knowledge. Experts also estimated the RR for the association between insufficient knowledge and NBT for HIV as 1.1-5.0. The RR estimate for the association between unsafe injection and NBT for HIV, adjusted for measured confounders, was 0.96 (95% confidence interval: 0.89,1.03). However, the RR estimate decreased to 0.82 (95% credible interval: 0.64, 0.99) after adjusting for insufficient knowledge as an unmeasured confounder. Conclusion: Our Bayesian approach that uses expert opinion to adjust for unmeasured confounders revealed that PWID who practice unsafe injection are more likely to be tested for HIV – an association that was not seen by conventional analysis.

Assessing the impact of unmeasured confounders for credible and reliable real-world evidence

Purpose: We review statistical methods for assessing the possible impact of bias due to unmeasured confounding in real world data analysis and provide detailed recommendations for choosing among the methods. Methods: By updating an earlier systematic review, we summarize modern statistical best practices for evaluating and correcting for potential bias due to unmeasuredconfounding in estimating causal treatment effect from non-interventional studies. Results: We suggest a hierarchical structure for assessing unmeasured confounding.First, for initial sensitivity analyses, we strongly recommend applying a recently developed method, the E-value, that is straightforward to apply and does not require prior knowledge or assumptions about the unmeasured confounder(s). When some such knowledge is available, the E-value could be supplemented by the rule-out or array method at this step. If these initial analyses suggest results may not be robust to unmeasured confounding, subsequent analyses could be conducted using more specialized statistical methods, which we categorize based on whether they requireaccess to external data on the suspected unmeasured confounder(s), internal data, or no data. Other factors for choosing the subsequent sensitivity analysis methods arealso introduced and discussed, including the types of unmeasured confounders and whether the subsequent sensitivity analysis is intended to provide a corrected causaltreatment effect. Conclusion: Various analytical methods have been proposed to address unmeasured confounding, but little research has discussed a structured approach to select appropriate methods in practice. In providing practical suggestions for choosing appropriate initial and, potentially, more specialized subsequent sensitivity analyses, we hopeto facilitate the widespread reporting of such sensitivity analyses in non-interventional studies. The suggested approach also has the potential to inform pre-specificationof sensitivity analyses before executing the analysis, and therefore increase the transparency and limit selective study reporting.

Bias factor, maximum bias and the E-value: insight and extended applications

Background Unmeasured confounding can bias the relationship between exposure and outcome. Sensitivity analyses generate bias-adjusted measures but these are not much used; this may change with the availability of the E-value (for evidence for causality in observational studies), appealing for its ease of calculation. However, as currently proposed, the E-value has some practical limitations that may reduce its use. Methods We first provide some insight into the relationship between two established measures for unmeasured confounding: ‘the bias factor’ and the maximum value this bias factor can take (‘the B bias’). These measures are the statistical foundation for the E-value. We use them to develop new E-value formulas for situations when it is not currently applicable such as e.g. when, not unusually, a negative relation between unmeasured confounder and outcome and a positive one with exposure are postulated. We also provide E-values on the odds ratio scale because, currently, even when using the odds ratio as the study measure in the calculation of E-value, the result is to be interpreted as a relative risk, which is somewhat inconvenient. Results The additional formulas for the E-value measure make it applicable in all possible scenarios defined by the combined directions between unmeasured confounder and both the exposure and outcome. In addition, E-value measures can now be interpreted as odds ratios if the observed results are reported on the same scale. Conclusions The E-value is part of newer sensitivity analyses methods for unmeasured confounding. We provide insight into its structure, underscoring its advantages and limitations, and expand its applications.