Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool

2018 ◽  
Vol 96 ◽  
pp. 54-62 ◽  
Author(s):  
Allison Gates ◽  
Ben Vandermeer ◽  
Lisa Hartling
Keyword(s):  
Machine Learning ◽  
Systematic Reviews ◽  
Risk Of Bias ◽  
Learning Tool ◽  
Cross Sectional ◽  
Bias Assessment ◽  
Machine Learning Tool

scholarly journals RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials

2015 ◽  
Vol 23 (1) ◽  
pp. 193-201 ◽  
Author(s):  
Iain J Marshall ◽  
Joël Kuiper ◽  
Byron C Wallace
Keyword(s):  
Machine Learning ◽  
Clinical Trials ◽  
Systematic Reviews ◽  
Risk Of Bias ◽  
Reasonable Accuracy ◽  
Bias Assessment ◽  
Cochrane Database ◽  
The Difference ◽  
Using Data ◽  
Unclear Risk

Abstract Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments. Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR. Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v 56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR). Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses.

scholarly journals Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on the Treatment of Acne Vulgaris: Cross-Sectional Analysis

10.2196/16978 ◽  
2020 ◽  
Vol 3 (1) ◽  
pp. e16978 ◽  
Author(s):  
Ryan Ottwell ◽  
Taylor C Rogers ◽  
J Michael Anderson ◽  
Austin Johnson ◽  
Matt Vassar
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
Acne Vulgaris ◽  
Cross Sectional Study ◽  
Primary Objective ◽  
Risk Of Bias ◽  
Selective Reporting ◽  
Cross Sectional ◽  
Bias Assessment ◽  
Meta Analyses

Background Spin is the misrepresentation of study findings, which may positively or negatively influence the reader’s interpretation of the results. Little is known regarding the prevalence of spin in abstracts of systematic reviews, specifically systematic reviews pertaining to the management and treatment of acne vulgaris. Objective The primary objective of this study was to characterize and determine the frequency of the most severe forms of spin in systematic review abstracts and to evaluate whether various study characteristics were associated with spin. Methods Using a cross-sectional study design, we searched PubMed and EMBASE for systematic reviews focusing on the management and treatment of acne vulgaris. Our search returned 316 studies, of which 36 were included in our final sample. To be included, each systematic review must have addressed either pharmacologic or nonpharmacologic treatment of acne vulgaris. These studies were screened, and data were extracted in duplicate by two blinded investigators. We analyzed systematic review abstracts for the nine most severe types of spin. Results Spin was present in 31% (11/36) of abstracts. A total of 12 examples of spin were identified in the 11 abstracts containing spin, with one abstract containing two instances of spin. The most common type of spin, selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention, was identified five times (5/12, 42%). A total of 44% (16/36) of studies did not report a risk of bias assessment. Of the 11 abstracts containing spin, six abstracts (55%) had not reported a risk of bias assessment or performed a risk of bias assessment but did not discuss it. Spin in abstracts was not significantly associated with a specific intervention type, funding source, or journal impact factor. Conclusions Spin is present in the abstracts of systematic reviews and meta-analyses covering the treatment of acne vulgaris. This paper raises awareness of spin in abstracts and emphasizes the importance of its recognition, which may lead to fewer incidences of spin in future studies.

scholarly journals Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

2021 ◽  
Vol 10 (1) ◽  
Author(s):  
Kevin E. K. Chai ◽  
Robin L. J. Lines ◽  
Daniel F. Gucciardi ◽  
Leo Ng
Keyword(s):  
Machine Learning ◽  
Systematic Reviews ◽  
Real World ◽  
New Technologies ◽  
Learning Tool ◽  
Level Of Evidence ◽  
Time Saving ◽  
Scoping Reviews ◽  
Abstract Screening ◽  
Machine Learning Tool

Abstract Background Systematic reviews and meta-analyses provide the highest level of evidence to help inform policy and practice, yet their rigorous nature is associated with significant time and economic demands. The screening of titles and abstracts is the most time consuming part of the review process with analysts required review thousands of articles manually, taking on average 33 days. New technologies aimed at streamlining the screening process have provided initial promising findings, yet there are limitations with current approaches and barriers to the widespread use of these tools. In this paper, we introduce and report initial evidence on the utility of Research Screener, a semi-automated machine learning tool to facilitate abstract screening. Methods Three sets of analyses (simulation, interactive and sensitivity) were conducted to provide evidence of the utility of the tool through both simulated and real-world examples. Results Research Screener delivered a workload saving of between 60 and 96% across nine systematic reviews and two scoping reviews. Findings from the real-world interactive analysis demonstrated a time saving of 12.53 days compared to the manual screening, which equates to a financial saving of USD 2444. Conservatively, our results suggest that analysts who scan 50% of the total pool of articles identified via a systematic search are highly likely to have identified 100% of eligible papers. Conclusions In light of these findings, Research Screener is able to reduce the burden for researchers wishing to conduct a comprehensive systematic review without reducing the scientific rigour for which they strive to achieve.

scholarly journals Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on the Treatment of Acne Vulgaris: Cross-Sectional Analysis (Preprint)

2019 ◽  
Author(s):  
Ryan Ottwell ◽  
Taylor C Rogers ◽  
J Michael Anderson ◽  
Austin Johnson ◽  
Matt Vassar
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
Acne Vulgaris ◽  
Cross Sectional Study ◽  
Primary Objective ◽  
Risk Of Bias ◽  
Selective Reporting ◽  
Cross Sectional ◽  
Bias Assessment ◽  
Meta Analyses

BACKGROUND Spin is the misrepresentation of study findings, which may positively or negatively influence the reader’s interpretation of the results. Little is known regarding the prevalence of spin in abstracts of systematic reviews, specifically systematic reviews pertaining to the management and treatment of acne vulgaris. OBJECTIVE The primary objective of this study was to characterize and determine the frequency of the most severe forms of spin in systematic review abstracts and to evaluate whether various study characteristics were associated with spin. METHODS Using a cross-sectional study design, we searched PubMed and EMBASE for systematic reviews focusing on the management and treatment of acne vulgaris. Our search returned 316 studies, of which 36 were included in our final sample. To be included, each systematic review must have addressed either pharmacologic or nonpharmacologic treatment of acne vulgaris. These studies were screened, and data were extracted in duplicate by two blinded investigators. We analyzed systematic review abstracts for the nine most severe types of spin. RESULTS Spin was present in 31% (11/36) of abstracts. A total of 12 examples of spin were identified in the 11 abstracts containing spin, with one abstract containing two instances of spin. The most common type of spin, <i>selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention,</i> was identified five times (5/12, 42%). A total of 44% (16/36) of studies did not report a risk of bias assessment. Of the 11 abstracts containing spin, six abstracts (55%) had not reported a risk of bias assessment or performed a risk of bias assessment but did not discuss it. Spin in abstracts was not significantly associated with a specific intervention type, funding source, or journal impact factor. CONCLUSIONS Spin is present in the abstracts of systematic reviews and meta-analyses covering the treatment of acne vulgaris. This paper raises awareness of spin in abstracts and emphasizes the importance of its recognition, which may lead to fewer incidences of spin in future studies.

scholarly journals Classification of apatite structures via topological data analysis: a framework for a ‘Materials Barcode’ representation of structure maps

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Scott Broderick ◽  
Ruhil Dongol ◽  
Tianmu Zhang ◽  
Krishna Rajan
Keyword(s):  
Machine Learning ◽  
Data Analysis ◽  
Crystal Chemistry ◽  
Persistent Homology ◽  
Hierarchical Classification ◽  
Topological Data Analysis ◽  
Learning Tool ◽  
Coordination Polyhedra ◽  
Machine Learning Tool ◽  
Topological Data

AbstractThis paper introduces the use of topological data analysis (TDA) as an unsupervised machine learning tool to uncover classification criteria in complex inorganic crystal chemistries. Using the apatite chemistry as a template, we track through the use of persistent homology the topological connectivity of input crystal chemistry descriptors on defining similarity between different stoichiometries of apatites. It is shown that TDA automatically identifies a hierarchical classification scheme within apatites based on the commonality of the number of discrete coordination polyhedra that constitute the structural building units common among the compounds. This information is presented in the form of a visualization scheme of a barcode of homology classifications, where the persistence of similarity between compounds is tracked. Unlike traditional perspectives of structure maps, this new “Materials Barcode” schema serves as an automated exploratory machine learning tool that can uncover structural associations from crystal chemistry databases, as well as to achieve a more nuanced insight into what defines similarity among homologous compounds.

scholarly journals Methodological quality of systematic reviews on treatments for depression: a cross-sectional study

2017 ◽  
Vol 27 (6) ◽  
pp. 619-627 ◽  
Author(s):  
V. C. H. Chung ◽  
X. Y. Wu ◽  
Y. Feng ◽  
R. S. T. Ho ◽  
S. Y. S. Wong ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Methodological Quality ◽  
A Priori ◽  
Cross Sectional Study ◽  
Risk Of Bias ◽  
Sectional Study ◽  
Pharmacological Treatments ◽  
Cross Sectional ◽  
Depression Treatments

Aims.Depression is one of the most common mental disorders and identifying effective treatment strategies is crucial for the control of depression. Well-conducted systematic reviews (SRs) and meta-analyses can provide the best evidence for supporting treatment decision-making. Nevertheless, the trustworthiness of conclusions can be limited by lack of methodological rigour. This study aims to assess the methodological quality of a representative sample of SRs on depression treatments.Methods.A cross-sectional study on the bibliographical and methodological characteristics of SRs published on depression treatments trials was conducted. Two electronic databases (the Cochrane Database of Systematic Reviews and the Database of Abstracts of Reviews of Effects) were searched for potential SRs. SRs with at least one meta-analysis on the effects of depression treatments were considered eligible. The methodological quality of included SRs was assessed using the validated AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using logistic regression analysis.Results.A total of 358 SRs were included and appraised. Over half of included SRs (n = 195) focused on non-pharmacological treatments and harms were reported in 45.5% (n = 163) of all studies. Studies varied in methods and reporting practices: only 112 (31.3%) took the risk of bias among primary studies into account when formulating conclusions; 245 (68.4%) did not fully declare conflict of interests; 93 (26.0%) reported an ‘a priori’ design and 104 (29.1%) provided lists of both included and excluded studies. Results from regression analyses showed: more recent publications were more likely to report ‘a priori’ designs [adjusted odds ratio (AOR) 1.31, 95% confidence interval (CI) 1.09–1.57], to describe study characteristics fully (AOR 1.16, 95% CI 1.06–1.28), and to assess presence of publication bias (AOR 1.13, 95% CI 1.06–1.19), but were less likely to list both included and excluded studies (AOR 0.86, 95% CI 0.81–0.92). SRs published in journals with higher impact factor (AOR 1.14, 95% CI 1.04–1.25), completed by more review authors (AOR 1.12, 95% CI 1.01–1.24) and SRs on non-pharmacological treatments (AOR 1.62, 95% CI 1.01–2.59) were associated with better performance in publication bias assessment.Conclusion.The methodological quality of included SRs is disappointing. Future SRs should strive to improve rigour by considering of risk of bias when formulating conclusions, reporting conflict of interests and authors should explicitly describe harms. SR authors should also use appropriate methods to combine the results, prevent language and publication biases, and ensure timely updates.

Sign in / Sign up

Export Citation Format

Share Document