Risk of bias assessment of test comparisons was uncommon in comparative accuracy systematic reviews: an overview of reviews

Abstract Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments. Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR. Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v 56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR). Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses.

Download Full-text

Toolkit of methodological resources to conduct systematic reviews

F1000Research ◽

10.12688/f1000research.22032.1 ◽

2020 ◽

Vol 9 ◽

pp. 82

Author(s):

Marta Roqué ◽

Laura Martínez-García ◽

Ivan Solà ◽

Pablo Alonso-Coello ◽

Xavier Bonfill ◽

...

Keyword(s):

Diagnostic Accuracy ◽

Systematic Reviews ◽

Assessment Tool ◽

Research Question ◽

Risk Of Bias ◽

Bias Assessment ◽

Assessment Development ◽

Stage Of Development ◽

Model Study ◽

Question Development

Background: Systematic reviews (SR) can be classified by type depending on the research question they are based on. This work identifies and describes the most relevant methodological resources to conduct high-quality reviews that answer clinical questions regarding prevalence, prognosis, diagnostic accuracy and efficacy of interventions. Methods: Methodological resources have been identified from literature searches and consulting guidelines from institutions that develop SRs. The selected resources are organized by type of SR, and stage of development of the review (formulation of the research question, development of the protocol, literature search, risk of bias assessment, synthesis of findings, assessment of the quality of evidence, and report of SR results and conclusions). Results: Although the different types of SRs are developed following the same steps, each SR type requires specific methods, differing in characteristics and complexity. The extent of methodological development varies by type of SR, with more solid guidelines available for diagnostic accuracy and efficacy of interventions SRs. This methodological toolkit describes the most up-to-date risk of bias instruments: Quality in Prognostic Studies (QUIPS) tool and Prediction model study Risk Of Bias Assessment Tool (PROBAST) for prognostic SRs, Quality assessment of diagnostic accuracy studies tool (QUADAS-2) for diagnostic accuracy SRs, Cochrane risk of bias tool (ROB-2) and Risk of bias in non-randomised studies of interventions studies tool (ROBINS-I) for efficacy of interventions SRs, as well as the latest developments on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Conclusions: This structured compilation of the best methodological resources for each type of SR may prove to be a very useful tool for those researchers that wish to develop SRs or conduct methodological research works on SRs.

Download Full-text

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

BMC Medical Research Methodology ◽

10.1186/s12874-020-01123-7 ◽

2020 ◽

Vol 20 (1) ◽

Author(s):

Ognjen Barcot ◽

Matija Boric ◽

Svjetlana Dosenovic ◽

Marija Cavar ◽

Antonia Jelicic Kadic ◽

...

Keyword(s):

Systematic Reviews ◽

Allocation Concealment ◽

Risk Of Bias ◽

Methodological Study ◽

Bias Assessment ◽

Surgical Trials ◽

Surgical Interventions ◽

Randomized Controlled ◽

Cochrane Reviews ◽

The Difference

Abstract Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

Download Full-text

Risk of bias assessment of systematic reviews regarding dental implant placement in smokers: An umbrella systematic review

Journal of Prosthetic Dentistry ◽

10.1016/j.prosdent.2017.12.026 ◽

2018 ◽

Vol 120 (2) ◽

pp. 198-203 ◽

Cited By ~ 2

Author(s):

Olavo B. de Oliveira-Neto ◽

Fabiano Timbo Barbosa ◽

Celio Fernando de Sousa-Rodrigues ◽

Fernando Jose Camello de Lima

Keyword(s):

Systematic Review ◽

Dental Implant ◽

Systematic Reviews ◽

Risk Of Bias ◽

Bias Assessment ◽

Implant Placement

Download Full-text

Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool

Journal of Clinical Epidemiology ◽

10.1016/j.jclinepi.2017.12.015 ◽

2018 ◽

Vol 96 ◽

pp. 54-62 ◽

Cited By ~ 10

Author(s):

Allison Gates ◽

Ben Vandermeer ◽

Lisa Hartling

Keyword(s):

Machine Learning ◽

Systematic Reviews ◽

Risk Of Bias ◽

Learning Tool ◽

Cross Sectional ◽

Bias Assessment ◽

Machine Learning Tool

Download Full-text

Toolkit of methodological resources to conduct systematic reviews

F1000Research ◽

10.12688/f1000research.22032.3 ◽

2020 ◽

Vol 9 ◽

pp. 82

Author(s):

Marta Roqué ◽

Laura Martínez-García ◽

Ivan Solà ◽

Pablo Alonso-Coello ◽

Xavier Bonfill ◽

...

Keyword(s):

Diagnostic Accuracy ◽

Systematic Reviews ◽

Assessment Tool ◽

Research Question ◽

Risk Of Bias ◽

Bias Assessment ◽

Assessment Development ◽

Stage Of Development ◽

Model Study ◽

Question Development

Background: Systematic reviews (SR) can be classified by type depending on the research question they are based on. This work identifies and describes the most relevant methodological resources to conduct high-quality reviews that answer health care questions regarding prevalence, prognosis, diagnostic accuracy and effects of interventions. Methods: Methodological resources have been identified from literature searches and consulting guidelines from institutions that develop SRs. The selected resources are organized by type of SR, and stage of development of the review (formulation of the research question, development of the protocol, literature search, risk of bias assessment, synthesis of findings, assessment of the quality of evidence, and report of SR results and conclusions). Results: Although the different types of SRs are developed following the same steps, each SR type requires specific methods, differing in characteristics and complexity. The extent of methodological development varies by type of SR, with more solid guidelines available for diagnostic accuracy and effects of interventions SRs. This methodological toolkit describes the most up-to-date risk of bias instruments: Quality in Prognostic Studies (QUIPS) tool and Prediction model study Risk Of Bias Assessment Tool (PROBAST) for prognostic SRs, Quality assessment of diagnostic accuracy studies tool (QUADAS-2) for diagnostic accuracy SRs, Cochrane risk of bias tool (ROB-2) and Risk of bias in non-randomised studies of interventions studies tool (ROBINS-I) for effects of interventions SRs, as well as the latest developments on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Conclusions: This structured compilation of the best methodological resources for each type of SR may prove to be a very useful tool for those researchers that wish to develop SRs or conduct methodological research works on SRs

Download Full-text

Toolkit of methodological resources to conduct systematic reviews

F1000Research ◽

10.12688/f1000research.22032.2 ◽

2020 ◽

Vol 9 ◽

pp. 82

Author(s):

Marta Roqué ◽

Laura Martínez-García ◽

Ivan Solà ◽

Pablo Alonso-Coello ◽

Xavier Bonfill ◽

...

Keyword(s):

Diagnostic Accuracy ◽

Systematic Reviews ◽

Assessment Tool ◽

Research Question ◽

Risk Of Bias ◽

Bias Assessment ◽

Assessment Development ◽

Stage Of Development ◽

Model Study ◽

Question Development

Background: Systematic reviews (SR) can be classified by type depending on the research question they are based on. This work identifies and describes the most relevant methodological resources to conduct high-quality reviews that answer health care questions regarding prevalence, prognosis, diagnostic accuracy and effects of interventions. Methods: Methodological resources have been identified from literature searches and consulting guidelines from institutions that develop SRs. The selected resources are organized by type of SR, and stage of development of the review (formulation of the research question, development of the protocol, literature search, risk of bias assessment, synthesis of findings, assessment of the quality of evidence, and report of SR results and conclusions). Results: Although the different types of SRs are developed following the same steps, each SR type requires specific methods, differing in characteristics and complexity. The extent of methodological development varies by type of SR, with more solid guidelines available for diagnostic accuracy and effects of interventions SRs. This methodological toolkit describes the most up-to-date risk of bias instruments: Quality in Prognostic Studies (QUIPS) tool and Prediction model study Risk Of Bias Assessment Tool (PROBAST) for prognostic SRs, Quality assessment of diagnostic accuracy studies tool (QUADAS-2) for diagnostic accuracy SRs, Cochrane risk of bias tool (ROB-2) and Risk of bias in non-randomised studies of interventions studies tool (ROBINS-I) for effects of interventions SRs, as well as the latest developments on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Conclusions: This structured compilation of the best methodological resources for each type of SR may prove to be a very useful tool for those researchers that wish to develop SRs or conduct methodological research works on SRs

Download Full-text

Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on the Treatment of Acne Vulgaris: Cross-Sectional Analysis

JMIR Dermatology ◽

10.2196/16978 ◽

2020 ◽

Vol 3 (1) ◽

pp. e16978 ◽

Cited By ~ 1

Author(s):

Ryan Ottwell ◽

Taylor C Rogers ◽

J Michael Anderson ◽

Austin Johnson ◽

Matt Vassar

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Acne Vulgaris ◽

Cross Sectional Study ◽

Primary Objective ◽

Risk Of Bias ◽

Selective Reporting ◽

Cross Sectional ◽

Bias Assessment ◽

Meta Analyses

Background Spin is the misrepresentation of study findings, which may positively or negatively influence the reader’s interpretation of the results. Little is known regarding the prevalence of spin in abstracts of systematic reviews, specifically systematic reviews pertaining to the management and treatment of acne vulgaris. Objective The primary objective of this study was to characterize and determine the frequency of the most severe forms of spin in systematic review abstracts and to evaluate whether various study characteristics were associated with spin. Methods Using a cross-sectional study design, we searched PubMed and EMBASE for systematic reviews focusing on the management and treatment of acne vulgaris. Our search returned 316 studies, of which 36 were included in our final sample. To be included, each systematic review must have addressed either pharmacologic or nonpharmacologic treatment of acne vulgaris. These studies were screened, and data were extracted in duplicate by two blinded investigators. We analyzed systematic review abstracts for the nine most severe types of spin. Results Spin was present in 31% (11/36) of abstracts. A total of 12 examples of spin were identified in the 11 abstracts containing spin, with one abstract containing two instances of spin. The most common type of spin, selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention, was identified five times (5/12, 42%). A total of 44% (16/36) of studies did not report a risk of bias assessment. Of the 11 abstracts containing spin, six abstracts (55%) had not reported a risk of bias assessment or performed a risk of bias assessment but did not discuss it. Spin in abstracts was not significantly associated with a specific intervention type, funding source, or journal impact factor. Conclusions Spin is present in the abstracts of systematic reviews and meta-analyses covering the treatment of acne vulgaris. This paper raises awareness of spin in abstracts and emphasizes the importance of its recognition, which may lead to fewer incidences of spin in future studies.

Download Full-text

Evidence-based occupational health and safety interventions: a comprehensive overview of reviews

BMJ Open ◽

10.1136/bmjopen-2019-032528 ◽

2019 ◽

Vol 9 (12) ◽

pp. e032528

Author(s):

Birgit Teufer ◽

Agnes Ebenberger ◽

Lisa Affengruber ◽

Christina Kien ◽

Irma Klerings ◽

...

Keyword(s):

Occupational Health ◽

Systematic Reviews ◽

Occupational Injuries ◽

Health And Safety ◽

Occupational Health And Safety ◽

Risk Of Bias ◽

Cochrane Library ◽

Evidence Based ◽

Comprehensive Overview ◽

Overview Of Reviews

ObjectivesOccupational injuries and diseases are a huge public health problem and cause extensive suffering and loss of productivity. Nevertheless, many occupational health and safety (OHS) guidelines are still not based on the best available evidence. In the last decade, numerous systematic reviews on behavioural, relational and mixed interventions to reduce occupational injuries and diseases have been carried out, but a comprehensive synopsis is yet missing. The aim of this overview of reviews is to provide a comprehensive basis to inform evidence-based decision-making about interventions in the field of OHS.MethodsWe conducted an overview of reviews. We searched MEDLINE (Ovid), the Cochrane Library (Wiley), epistemonikos.org and Scopus (Elsevier) for relevant systematic reviews published between January 2008 and June 2018. Two authors independently screened abstracts and full-text publications and determined the risk of bias of the included systematic reviews with the ROBIS (Risk of Bias in Systematic Reviews) tool.ResultsWe screened 2287 abstracts and 200 full-texts for eligibility. Finally, we included 25 systematic reviews with a low risk of bias for data synthesis and analysis. We identified systematic reviews on the prevention of occupational injuries, musculoskeletal, skin and lung diseases, occupational hearing impairment and interventions without specific target diseases. Several interventions led to consistently positive results on individual diseases; other interventions did not show any effects, or the studies are contradictory. We provide detailed results on all included interventions.DiscussionTo our knowledge, this is the first comprehensive overview of behavioural, relational and mixed interventions and their effectiveness in preventing occupational injuries and diseases. It provides policymakers with an important basis for making evidence-based decisions on interventions in this field.PROSPERO registration numberCRD42018100341

Download Full-text

Evaluation of Industry Relationships Among Authors of Systematic Reviews and Meta-analyses Regarding Ménières Disease

Annals of Otology Rhinology & Laryngology ◽

10.1177/00034894211051822 ◽

2021 ◽

pp. 000348942110518

Author(s):

David Wenger ◽

Ross Nowlin ◽

Austin L. Johnson ◽

Michael Anderson ◽

Michael Weaver ◽

...

Keyword(s):

High Risk ◽

Systematic Reviews ◽

Conflicts Of Interest ◽

Cochrane Collaboration ◽

Risk Of Bias ◽

Inclusion Criteria ◽

Disease Treatment ◽

Bias Assessment ◽

Meta Analyses ◽

The Cochrane Collaboration

Objectives: To quantify the presence of conflicts of interest (COI) in SRs and MAs of Ménières disease treatment and identify any related secondary characteristics of these articles. Methods: A search was conducted on May 28, 2020 to search MEDLINE and Embase databases for SRs or MAs pertaining to Ménières disease published between September 1, 2016 and June 2, 2020. A risk of bias assessment was performed using the Cochrane Collaboration risk of bias assessment criteria. Results: A total of 13 systematic reviews conducted by 49 authors met the inclusion criteria. Of the 49 authors, 7 (14.3%) were found to have some form of COI. Of these 7 authors, 1 (14.3%) completely disclosed all COI within the SR, 1 (14.3%) disclosed one or more COI but were found to have an additional undisclosed COI, and 5 (71.4%) were found to have only undisclosed COI. One of 2 industry funded SRs (50%) had a high risk of bias, and 1 (50%) of the non-industry sponsored SRs were found to have a high risk of bias. Conclusions: Overall authors of SRs pertaining to Ménières disease appear to be properly disclosing COI at higher rates than other fields of medicine; however, further room for improvement has been noted.

Download Full-text