Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength

IntroductionAssessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Suprathreshold perceptual decisions constrain models of confidence

Perceptual confidence is an important internal signal about the certainty of our decisions and there is a substantial debate on how it is computed. We highlight three confidence metric types from the literature: observers either use 1) the full probability distribution to compute probability correct (Probability metrics), 2) point estimates from the perceptual decision process to estimate uncertainty (Evidence-Strength metrics), or 3) heuristic confidence from stimulus-based cues to uncertainty (Heuristic metrics). These metrics are rarely tested against one another, so we examined models of all three types on a suprathreshold spatial discrimination task. Observers were shown a cloud of dots sampled from a dot generating distribution and judged if the mean of the distribution was left or right of centre. In addition to varying the horizontal position of the mean, there were two sensory uncertainty manipulations: the number of dots sampled and the spread of the generating distribution. After every two perceptual decisions, observers made a confidence forced-choice judgement whether they were more confident in the first or second decision. Model results showed that observers were on average best-fit by a Heuristic model that used dot cloud position, spread, and number of dots as cues. However, almost half of the observers were best-fit by an Evidence-Strength model that uses the distance between the discrimination criterion and a point estimate, scaled according to sensory uncertainty, to compute confidence. This signal-to-noise ratio model outperformed the standard unscaled distance from criterion model favoured by many researchers and suggests that this latter simple model may not be suitable for mixed-difficulty designs. An accidental repetition of some sessions also allowed for the measurement of confidence agreement for identical pairs of stimuli. This N-pass analysis revealed that human observers were more consistent than their best-fitting model would predict, indicating there are still aspects of confidence that are not captured by our model. As such, we propose confidence agreement as a useful technique for computational studies of confidence. Taken together, these findings highlight the idiosyncratic nature of confidence computations for complex decision contexts and the need to consider different potential metrics and transformations in the confidence computation.

Effectiveness of Health Interventions towards HIV/AIDS in Developing Countries: A Systematic Review

Background: Plentiful development has been achieved in interventions for the prevention of HIV. Although, progression of prevention programs based on evidence – informed methods that interpret the effectiveness of these approaches in population is still a challenge. In developing countries, not many interventions are implemented for reduction of HIV burden. The single most important identified problem is lack of demand, supply, and adherence approaches. In current systemic review, recent evidence for the prevention of HIV in a cascade manner is described to see status of current interventions and further needs for improvements. Methodology: Systemic reviews regarding effectiveness on interventions of HIV prevention were searched. Primary studies were identified from eligible review that evaluated one of following factors: prevalence of HIV, incidence of HIV, testing uptake of HIV and use of condom. Interventions were categorized that pursued demand for prevention of HIV, improvement in supply for preventive approaches, support related to preventive behaviors or prevent HIV directly. A rating was assigned for each intervention based on evidence strength or randomized controlled trials. Results: Out of 91 eligible reviews, 264 primary studies were included in this review. Primary studies related to direct mechanisms of prevention that showed strong data for circumcision and effectiveness of pre – exposure prophylaxis. Evidence implies that interventions related to increased supply of preventive methods including clean needles or condoms can be operative. Interventions related to demand – side and adherence approaches were less clear with some studies showing effectiveness. Quality evidence was assessed among various categories. Various interventions showed supportive outcomes and results. In our findings, it was observed that difference between behavioral and structural has not evidently distinguished the interventions. Conclusion: Growing data is present for the support of effectiveness of products, behaviors, and procedures for prevention of HIV. In developing countries, negligible data is present for implementation of such approaches on community level. Interventions will be required for transforming this evidence to produce impact on population. It will empower the demand for prevention of HIV, supply of preventive technologies and utilization of preventive approaches against HIV. The findings can be eye opener to see actual burden of HIV and their implanted interventions and can be useful to design further intervention programs in future.

How to report E-values for meta-analyses: Recommended improvements and additions to the new GRADE approach

In a recent concept paper (Verbeek et al., 2021), the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group provides a preliminary proposal to improve its existing guidelines for assessing sensitivity to uncontrolled confounding in meta-analyses of nonrandomized studies. The new proposal centers on reporting the E-value for the meta-analytic mean and on comparing this E-value to a measured “reference confounder” to determine whether residual uncontrolled confounding in the meta-analyzed studies could or could not plausibly explain away the meta-analytic mean. Although we agree that E-value analogs for meta-analyses could be an informative addition to future GRADE guidelines, we suggest improvements to the Verbeek et al. (2021)’s specific proposal regarding: (1) their interpretation of comparisons between the E-value and the strengths of associations of a reference confounder; (2) their characterization of evidence strength in meta-analyses in terms of only the meta-analytic mean; and (3) the possibility of confounding bias that is heterogeneous across studies.

Methods to Address Confounding and Other Biases in Meta-Analyses: Review and Recommendations

Meta-analyses contribute critically to cumulative science, but they can produce misleading conclusions if their constituent primary studies are biased, for example by unmeasured confounding in nonrandomized studies. We provide practical guidance on how meta-analysts can address confounding and other biases that affect studies’ internal validity, focusing primarily on sensitivity analyses that help quantify how biased the meta-analysis estimates might be. We review a number of sensitivity analysis methods to do so, especially recent developments that are straightforward to implement and interpret and that use somewhat less stringent statistical assumptions than do earlier methods. We give recommendations for how these newer methods could be applied in practice and illustrate using a previously published meta-analysis. Sensitivity analyses can provide informative quantitative summaries of evidence strength, and we suggest reporting them routinely in meta-analyses of potentially biased studies. This recommendation in no way diminishes the importance of defining study eligibility criteria that reduce bias and of characterizing studies’ risks of bias qualitatively. Expected final online publication date for the Annual Review of Public Health, Volume 43 is April 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

The Global Virome in One Network (VIRION): an atlas of vertebrate-virus associations

Data cataloguing viral diversity on Earth have been fragmented across sources, disciplines, formats, and various degrees of open collation, posing challenges for research on macroecology, evolution, and public health. Here, we solve this problem by establishing a dynamically-maintained database of vertebrate-virus associations, called The Global Virome in One Network (VIRION). The VIRION database has been assembled through both reconciliation of static datasets and integration of dynamically-updated databases. These data sources are all harmonized against one taxonomic backbone, including metadata on host and virus taxonomic validity and higher classification; additional metadata on sampling methodology and evidence strength are also available in a harmonized format. In total, the VIRION database is the largest open-source, open-access database of its kind, with roughly half a million unique records that include 9,521 resolved virus "species" (of which 1,661 are ICTV ratified), 3,692 resolved vertebrate host species, and 23,147 unique interactions between taxonomically-valid organisms. Together, these data cover roughly a quarter of mammal diversity, a tenth of bird diversity, and ˜6% of the estimated total diversity of vertebrates, and a much larger proportion of their virome than any previous database. We show how these data can be used to test hypotheses about microbiology, ecology, and evolution, and make suggestions for best practices that address the unique mix of evidence that coexists in these data.

The effects of evidence-based expert testimony on perceptions of child sexual abuse involving recantation

Child sexual abuse (CSA) cases involving recantation invoke concerns about children’s reliability. Expert testimony can help explain the complexities of these cases. Experts have historically relied on Child Sexual Abuse Accommodation Syndrome (CSAAS), yet this is not science-based. In a CSA case involving recantation, how would evidence-based testimony affect perceptions of child credibility when compared to CSAAS? Across 2 studies, we test the effects of expert testimony based on evidence-based science, nonscientific evidence, and experience-based evidence on outcomes in CSA cases involving recantation. Evidence-based testimony led to higher perceptions of credibility and scientific rigor of the evidence when compared to CSAAS testimony. Evidence-based testimony also led to more guilty verdicts when compared to the control. In sum, jurors had some ability to detect evidence strength, such that evidence-based expert testimony was superior to CSAAS testimony in many respects, and consistently superior to experience-based testimony in these cases.