Machine Learning Reduced Workload for the Cochrane COVID-19 Study Register: Development and Evaluation of the Cochrane COVID-19 Study Classifier

Background: This study developed, calibrated, and evaluated a machine learning (ML) classifier designed to reduce study identification workload inmaintainingthe Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies.Methods: A ML classifier for retrieving COVID-19 research studies (the “Cochrane COVID-19 Study Classifier”) was developedusing a data set of title-abstract records ‘included’ in, or‘excluded’ from,the CCSR up to18th October 2020, manually labelled byinformation and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from,theCCSRbetween 19th October and 2ndDecember 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between4thand 19thJanuary 2021.Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6,005 of which were ‘included’ in the CCSR) and the classifier hada precision of 0.52in this dataset atthe target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2,285 (98.9%) of 2,310eligible study reports but missed 25 (1%), with aprecision of 0.638 and a netscreening workload reduction of 24.1% (1,113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifierreduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.

Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow

Background Current text mining tools supporting abstract screening in systematic reviews are not widely used, in part because they lack sensitivity and precision. We set out to develop an accessible, semi-automated “workflow” to conduct abstract screening for systematic reviews and other knowledge synthesis methods. Methods We adopt widely recommended text-mining and machine-learning methods to (1) process title-abstracts into numerical training data; and (2) train a classification model to predict eligible abstracts. The predicted abstracts are screened by human reviewers for (“true”) eligibility, and the newly eligible abstracts are used to identify similar abstracts, using near-neighbor methods, which are also screened. These abstracts, as well as their eligibility results, are used to update the classification model, and the above steps are iterated until no new eligible abstracts are identified. The workflow was implemented in R and evaluated using a systematic review of insulin formulations for type-1 diabetes (14,314 abstracts) and a scoping review of knowledge-synthesis methods (17,200 abstracts). Workflow performance was evaluated against the recommended practice of screening abstracts by 2 reviewers, independently. Standard measures were examined: sensitivity (inclusion of all truly eligible abstracts), specificity (exclusion of all truly ineligible abstracts), precision (inclusion of all truly eligible abstracts among all abstracts screened as eligible), F1-score (harmonic average of sensitivity and precision), and accuracy (correctly predicted eligible or ineligible abstracts). Workload reduction was measured as the hours the workflow saved, given only a subset of abstracts needed human screening. Results With respect to the systematic and scoping reviews respectively, the workflow attained 88%/89% sensitivity, 99%/99% specificity, 71%/72% precision, an F1-score of 79%/79%, 98%/97% accuracy, 63%/55% workload reduction, with 12%/11% fewer abstracts for full-text retrieval and screening, and 0%/1.5% missed studies in the completed reviews. Conclusion The workflow was a sensitive, precise, and efficient alternative to the recommended practice of screening abstracts with 2 reviewers. All eligible studies were identified in the first case, while 6 studies (1.5%) were missed in the second that would likely not impact the review’s conclusions. We have described the workflow in language accessible to reviewers with limited exposure to natural language processing and machine learning, and have made the code available to reviewers.

Dental Workload Reduction during First SARS-CoV-2/COVID-19 Lockdown in Germany: A Cross-Sectional Survey

An observational cross-sectional survey was planned to analyze the weekly workload reduction of German dentists during lockdown due to the global COVID-19 pandemic. Participants were predominantly members of the Free Association of German Dentists and filled in an online questionnaire. The questionnaire was sent to a total of 9416 dentists, with a response rate of 27.98% (n = 2635). Respondents were divided into seven macro areas by gross domestic product. Nearly two-thirds of dentists (65.16%) reported a reduction in their practice workload of more than 50% compared to the pre-pandemic period with statistically significant differences between German macro areas (p < 0.01). Weekly workload was reduced during the lockdown in 93.00% of study participants, while 55.33% dental care centers with multiple employed dentists under the direction of a non-dentist general manager had only a 40% reduction in weekly workload compared to a solo practice or a practice of a dentist with an employed dentist (30.24% and 28.39%, respectively). Dentists in Germany drastically reduced their practice activity during the first wave of the COVID-19 lockdown, both in rural and urban areas. Short, medium, and long-term effects of the pandemic on dental practices, dental staff as well as patient care need to be further investigated.

Validation of a Semiautomated Surveillance Algorithm for Deep Surgical Site Infections After Primary Hip or Knee Arthroplasty

Background: Surgical site infections (SSIs) complicate ~2% of primary total hip (THAs) or total knee arthroplasties (TKAs). Accurate and timely identification through surveillance is essential for targeted implementation and monitoring of preventive interventions. Electronic health records (EHR) facilitate (semi)-automated surveillance and enable upscaling. A validated algorithm is a prerequisite for broader implementation of semiautomated surveillance. Objectives: To validate a previously published algorithm for semiautomated surveillance of deep SSI after THA or TKA in 4 independent regional Dutch hospitals. The algorithm was developed and implemented in the University Medical Centre Utrecht and relies on retrospective routine care data. Methods: For this multicenter retrospective cohort study, the following data required for the algorithm were extracted from the EHR from all patients under THA and TKA surveillance: microbiology results, antibiotics, (re)admissions, and surgical procedures within the 120 days following the primary surgery. Patients were retrospectively classified with a low or high probability of having developed a deep SSI after THA or TKA, according to the algorithm. Sensitivity, positive predictive value (PPV), and workload reduction (defined as the proportion of manuals requiring review) were calculated compared to the traditional (manual) surveillance results, as reported to the national surveillance PREZIES. Discrepancy analyses were performed to understand algorithm results. Results: Data from 8,378 total THA and TKA surgeries (deep SSI n = 95, 1.1%) performed between 2012 and 2018 were extracted by 4 hospitals (Table 1). Sensitivity ranged across centers from 93.6% to 100%, with a PPV from 55.8% to 72.2%. In all hospitals, the algorithm resulted in >98% workload reduction. Cases missed by the algorithm could be explained by incomplete data extraction. Conclusions: This study shows that the surveillance algorithm performance is good in general Dutch hospitals. Broader implementation of this semiautomated surveillance for SSIs after THA or TKA may be possible in the near future and will result in a substantial workload reduction.Funding: This work was supported by the Regional Healthcare Network Antibiotic Resistance Utrecht with a subsidy of the Dutch Ministry of Health, Welfare and Sport (grant number 326835).Disclosures: None

Evaluation of Novel Concepts for Takeover Control Using Electronically Coupled Sidesticks

The state of the of art in flight control systems geared toward dual-pilot helicopters is the use of active inceptor systems to replace the traditional mechanical linkage between pilot and copilot inceptors. This work investigates the introduction of priority functions, which act to actively decouple inceptors in one control station. This approach has the potential to assist pilots to take over control in low-level flight and aid to mitigate loss-of-control accidents that occur in such conditions. Takeover control maneuvers are tested in a dual-pilot helicopter simulation environment to evaluate two inceptor decoupling methods, namely a priority pushbutton (manual) and a priority force threshold (automatic). Results indicate that the takeover maneuvers were successfully performed in low-level flight without over control (inaccurate control inputs) when using both priority functions. The priority functions led to a workload reduction when compared to a benchmark configuration without inceptor decoupling. Positive ratings in usefulness and satisfaction scales indicate pilot acceptance of the priority functions tested.