The Revolution Will Be Hard to Evaluate: How Co-Occurring Policy Changes Affect Research on the Health Effects of Social Policies

Extensive empirical health research leverages variation in the timing and location of policy changes as quasi-experiments. Multiple social policies may be adopted simultaneously in the same locations, creating co-occurrence which must be addressed analytically for valid inferences. The pervasiveness and consequences of co-occurring policies have received limited attention. We analyzed a systematic sample of 13 social policy databases covering diverse domains including poverty, paid family leave, and tobacco. We quantified policy co-occurrence in each database as the fraction of variation in each policy measure across different jurisdictions and times that could be explained by co-variation with other policies (R2). We used simulations to estimate the ratio of the variance of effect estimates under the observed policy co-occurrence to variance if policies were independent. Policy co-occurrence ranged from very high for state-level cannabis policies to low for country-level sexual minority rights policies. For 65% of policies, greater than 90% of the place-time variation was explained by other policies. Policy co-occurrence increased the variance of effect estimates by a median of 57-fold. Co-occurring policies are common and pose a major methodological challenge to rigorously evaluating health effects of individual social policies. When uncontrolled, co-occurring policies confound one another, and when controlled, resulting positivity violations may substantially inflate the variance of estimated effects. Tools to enhance validity and precision for evaluating co-occurring policies are needed.

Is the Way Forward to Step Back? Documenting the Frequency with which Study Goals are Misaligned with Study Methods and Interpretations in the Epidemiologic Literature

In any research study, there is an underlying research process that should begin with a clear articulation of the study’s goal. The study’s goal drives this process; it determines many study features including the estimand of interest, the analytic approaches that can be used to estimate it, and which coefficients, if any, should be interpreted. “Misalignment” can occur in this process when analytic approaches and/or interpretations do not match the study’s goal; misalignment is potentially more likely to arise when study goals are ambiguously framed. This study documented misalignment in the observational epidemiologic literature and explored how the framing of study goals contributes to its occurrence. The following misalignments were examined: 1) use of an inappropriate variable selection approach for the goal (a “goal-methods” misalignment) and 2) interpretation of coefficients of variables for which causal considerations were not made (e.g., Table 2 Fallacy, a “goal-interpretation” misalignment). A random sample of 100 articles published 2014-2018 in the top 5 general epidemiology journals were reviewed. Most reviewed studies were causal, with either explicitly stated (13/103, 13%) or associationally-framed (71/103, 69%) aims. Full alignment of goal-methods-interpretations was infrequent (9/103, 9%), although clearly causal studies (5/13, 38%) were more often fully aligned than seemingly causal ones (3/71, 4%). Goal-methods misalignments were common (34/103, 33%), but most frequently, methods were insufficiently reported to draw conclusions (47/103, 46%). Goal-interpretations misalignments occurred in 31% (32/103) of studies and occurred less often when the methods were aligned (2/103, 2%) compared with when the methods were misaligned (13/103, 13%).

Evolution and Reproducibility of Simulation Modeling in Epidemiology and Health Policy over Half a Century

Simulation models are increasingly used to inform epidemiological studies and health policy, yet there is great variation in their transparency and reproducibility. This review provides an overview of applications of simulation models in health policy and epidemiology, analyzes the use of best reporting practices, and assesses the reproducibility of the models using predefined, categorical criteria. 1,613 studies were identified and analyzed. We found an exponential growth in the number of studies over the past half century, with the highest growth in dynamic modeling approaches. The largest subset of studies is focused on disease policy models (70%), within which pathological conditions, viral diseases, neoplasms, and cardiovascular diseases account for one-third of the articles. Nearly half of the studies do not report the details of their models. We also provide in depth analysis of modeling best practices, reporting quality and reproducibility for a subset of 100 articles (50 highly cited and 50 random). Only seven of 26 in-depth evaluation criteria were satisfied by more than 80% of samples. We identify areas for increased application of simulation modeling and opportunities to enhance the rigor and documentation in the conduct and reporting of simulation modeling in epidemiology and health policy.

Burden of Antimicrobial Resistance: Compared to What?

There has been an increased focus on the public health burden of antimicrobial resistance (AMR). This raises conceptual challenges such as determining how much harm multi-drug resistant organisms do compared to what, or how to establish the burden. In this viewpoint we will present a counterfactual framework and provide guidance to harmonize methodologies and optimize study quality. In AMR burden studies, two counterfactual approaches have been applied; the harm of drug-resistant infections relative to the harm of the same, drug-susceptible, infections (susceptible-infection counterfactual) and the total harm of drug-resistant infections relative to a situation where such infections were prevented (no-infection counterfactual). We propose to use an intervention-based causal approach to determine the most appropriate counterfactual. We show that intervention scenarios, species of interest, and types of infections influence the choice of counterfactual. We recommend using purpose-designed cohort studies to apply this counterfactual framework, whereby the selection of cohorts (patients with drug-resistant, drug-susceptible and no-infection) should be based on matching on time to infection through exposure density sampling to avoid biased estimates. Application of survival methods is preferred, considering competing events. In conclusion, we advocate to estimate the burden of AMR using the no-infection and susceptible-infection counterfactuals. The resulting numbers will provide policy-relevant information about the upper and lower bound of future interventions designed to control AMR. The counterfactuals should be applied in cohort studies, whereby selection of the unexposed cohorts should be based on exposure density sampling, applying methods avoiding time-dependent bias and confounding.