Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength

IntroductionAssessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

The quantitative impact of joint peer review with a specialist radiologist in head and neck cancer radiotherapy planning

Objectives: Radiologist input in peer review of head and neck radiotherapy has been introduced as a routine departmental approach. The aim was to evaluate this practice and to quantitatively analyse the changes made. Methods: Patients treated with radical-dose radiotherapy between August–November 2020 were reviewed. The incidence of major and minor changes, as defined by The Royal College of Radiologists guidance, was prospectively recorded. The amended radiotherapy volumes were compared with the original volumes using Jaccard Index (JI) to assess conformity; Geographical Miss Index (GMI) for under contouring; and Hausdorff Distance (HD) between the volumes. Results: In total 73 out of 87 (84%) patients were discussed. Changes were recommended in 38 (52%) patients: 30 had ≥1 major change, eight had minor changes only. There were 99 amended volumes: The overall median JI, GMI and HD was 0.91 (interquartile range [IQR]=0.80–0.97), 0.06 (IQR = 0.02–0.18) and 0.42 cm (IQR = 0.20–1.17 cm) respectively. The nodal gross-tumour-volume (GTVn) and therapeutic high-dose nodal clinical-target-volume (CTVn) had the biggest magnitude of changes: The median JI, GMI and HD of GTVn was 0.89 (IQR = 0.44–0.95), 0.11 (IQR = 0.05–0.51), 3.71 cm (IQR = 0.31–6.93 cm); high-dose CTVn was 0.78 (IQR = 0.59–0.90), 0.20 (IQR = 0.07–0.31) and 3.28 cm (IQR = 1.22–6.18 cm) respectively. There was no observed difference in the quantitative indices of the 85 ‘major’ and 14 ‘minor’ volumes (p = 0.5). Conclusions: Routine head and neck radiologist input in radiotherapy peer review is feasible and can help avoid gross error in contouring. Advances in knowledge: The major and minor classification may benefit from differentiation with quantitative indices but requires correlation from clinical outcomes.

Quantitative Impact Analysis of Climate Change on Residents’ Health Conditions with Improving Eco-Efficiency in China: A Machine Learning Perspective

Climate change affects public health, and improving eco-efficiency means reducing the various pollutants that are the result of economic activities. This study provided empirical evidence of the quantitative impact analysis of climate change on the health conditions of residents across China due to improvements that have been made to eco-efficiency. First, the indicators that were collected present adequate graphical trends and regional differences with a priori evidence about their relationships to each other; second, the present study applied Sensitivity Evaluation with Support Vector Machines (SE-SVM) to Chinese provincial panel data, taking the Visits to Hospitals, Outpatients with Emergency Treatment, and Number of Inpatients as proxy variables for the health conditions of the residents in each area and temperature, humidity, precipitation, and sunshine as the climate change variables, simultaneously incorporating the calculated eco-efficiency with six controlling indicators; third, we compared in-sample forecasting to acquire the optimal model in order to conduct elasticity analysis. The results showed that (1) temperature, humidity, precipitation, and sunshine performed well in forecasting the health conditions of the residents and that climate change was a good forecaster for resident health conditions; (2) from the national perspective, climate change had a positive relationship with Visits to Hospitals and Outpatients with Emergency Treatment but a negative relationship with the Number of Inpatients; (3) An increase in regional eco-efficiency of 1% increase the need for Visits to Hospitals and Outpatients with Emergency Treatment by 0.2242% and 0.2688%, respectively, but decreased the Number of Inpatients by 0.6272%; (4) increasing the regional eco-efficiency did not show any positive effects for any individual region because a variety of local activities, resource endowment, and the level of medical technology available in each region played different roles. The main findings of the present study are helpful for decision makers who are trying to optimize policy formulation and implementation measures in the cross-domains of economic, environmental, and public health.

Food Waste along the Food Chain in Romania: An Impact Analysis

Food waste is a hot topic around the world due to the significant environmental challenge it poses. The study aims to assess the impact of food waste on the food chain at the national level. The data were obtained from quantitative impact studies, carried out in a project funded by the Ministry of Agriculture and Sustainable Development, “Methods to reduce food waste on the agri-food chain, at national level, to prevent and reduce socio-economic impact, until 2030”. A total of 852 companies were interviewed, with a turnover of almost 6.5 billion euro and a number of over 69 thousand employees, including 273 primary production enterprises, 270 food processing units, 171 distribution/retail units, and 138 HoReCa units.

Impact assessment methodology for connected and automated vehicles

Connected and automated vehicles have become more common in recent years, increasing the need to assess their societal level impacts. In this paper a methodology is presented to explore and define these impacts as a starting point for quantitative impact assessment. The many interrelations between impacts increases the complexity of obtaining a complete overview. Therefore, a structured approach is used, which shows many similarities with the modelling of causal-loop-diagrams. Feedback loops between impacts are taken into account at an early stage and both literature review and expert interviews are used to produce a holistic overview of impacts. The methodology was developed and applied in the European H2020 project LEVITATE. Results from the qualitative assessments in this project are presented and further steps needed to perform a quantitative evaluation of the impacts are discussed.

Detailed dual-Doppler structure of Kelvin-Helmholtz waves from an airborne profiling radar over complex terrain. Part II: Evidence for precipitation enhancement from observations and modeling

Kelvin-Helmholtz (KH) waves are a frequent source of turbulence in stratiform precipitation systems over mountainous terrain. KH waves introduce large eddies into otherwise laminar flow, with updrafts and downdrafts generating small-scale turbulence. When they occur in-cloud, such dynamics influence microphysical processes that impact precipitation growth and fallout. Part I of this paper used dual-Doppler, 2D wind and reflectivity measurements from an airborne cloud radar to demonstrate the occurrence of KH waves in stratiform orographic precipitation systems and identified four mechanisms for triggering KH waves. In Part II, we use similar observations to explore the effects of KH wave updrafts and turbulence on cloud microphysics. Measurements within KH wave updrafts reveal the production of liquid water in otherwise ice-dominated clouds, which can contribute to snow generation or enhancement via depositional and accretional growth. Fallstreaks beneath KH waves contain higher ice water content, composed of larger and more numerous ice particles, suggesting that KH waves and associated turbulence may also increase ice nucleation.A Large-Eddy Simulation (LES), designed to model the microphysical response to the KH wave eddies in mixed phase cloud, shows that depositional and accretional growth can be enhanced in KH waves, resulting in more precipitation when compared to a baseline simulation. While sublimation and evaporation occur in KH downdrafts, persistent supersaturation with respect to ice allows for net increase in ice mass. These modeling results and observations suggest that KH waves embedded in mixed-phase stratiform clouds may increase precipitation, although the quantitative impact remains uncertain.