scholarly journals Assessments of risk of bias in systematic reviews of observational nutritional epidemiologic studies are often not appropriate or comprehensive: a methodological study

2021 ◽  
pp. e000248
Author(s):  
Dena Zeraatkar ◽  
Alana Kohut ◽  
Arrti Bhasin ◽  
Rita E Morassut ◽  
Isabella Churchill ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Epidemiologic Studies ◽  
Risk Of Bias ◽  
Selective Reporting ◽  
Methodological Study ◽  
Essential Component ◽  
Cochrane Database

BackgroundAn essential component of systematic reviews is the assessment of risk of bias. To date, there has been no investigation of how reviews of non-randomised studies of nutritional exposures (called ‘nutritional epidemiologic studies’) assess risk of bias.ObjectiveTo describe methods for the assessment of risk of bias in reviews of nutritional epidemiologic studies.MethodsWe searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews (Jan 2018–Aug 2019) and sampled 150 systematic reviews of nutritional epidemiologic studies.ResultsMost reviews (n=131/150; 87.3%) attempted to assess risk of bias. Commonly used tools neglected to address all important sources of bias, such as selective reporting (n=25/28; 89.3%), and frequently included constructs unrelated to risk of bias, such as reporting (n=14/28; 50.0%). Most reviews (n=66/101; 65.3%) did not incorporate risk of bias in the synthesis. While more than half of reviews considered biases due to confounding and misclassification of the exposure in their interpretation of findings, other biases, such as selective reporting, were rarely considered (n=1/150; 0.7%).ConclusionReviews of nutritional epidemiologic studies have important limitations in their assessment of risk of bias.

scholarly journals RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials

2015 ◽  
Vol 23 (1) ◽  
pp. 193-201 ◽  
Author(s):  
Iain J Marshall ◽  
Joël Kuiper ◽  
Byron C Wallace
Keyword(s):  
Machine Learning ◽  
Clinical Trials ◽  
Systematic Reviews ◽  
Risk Of Bias ◽  
Reasonable Accuracy ◽  
Bias Assessment ◽  
Cochrane Database ◽  
The Difference ◽  
Using Data ◽  
Unclear Risk

Abstract Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments. Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR. Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v 56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR). Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses.

scholarly journals Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Ognjen Barcot ◽  
Matija Boric ◽  
Svjetlana Dosenovic ◽  
Marija Cavar ◽  
Antonia Jelicic Kadic ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Allocation Concealment ◽  
Risk Of Bias ◽  
Methodological Study ◽  
Bias Assessment ◽  
Surgical Trials ◽  
Surgical Interventions ◽  
Randomized Controlled ◽  
Cochrane Reviews ◽  
The Difference

Abstract Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

How do Systematic Reviews of Acupuncture for Pain Relief Incorporate Risk of Bias Assessments into the Synthesis? A Methodological Study

2016 ◽  
Vol 34 (2) ◽  
pp. 84-89 ◽  
Author(s):  
Xiao-qian Li ◽  
Kun-ming Tao ◽  
Ge-liang Yang ◽  
Qing-hui Zhou ◽  
Chang-quan Ling
Keyword(s):  
Pain Relief ◽  
Systematic Reviews ◽  
Internal Validity ◽  
Final Analysis ◽  
Risk Of Bias ◽  
Data Synthesis ◽  
Cochrane Database ◽  
Threshold Score ◽  
Domain Level ◽  
Appraisal Tool

Objective By reviewing the assessment of internal validity in relevant systematic reviews (SRs), the aim of this study was to identify how critical appraisals of risk of bias (RoB) inform the synthesis of evidence in SRs of acupuncture for pain relief. Methods SRs were searched in Medline, EMBASE, and the Cochrane Database of SRs from their inception to 30 December 2014. Only SRs of acupuncture for pain relief were included. Basic information, types of RoB appraisal tool, whether or not there was domain-level assessment of RoB, whether or not the reviews ranked studies by RoB, plus whether or not (and, if so, how) RoB appraisal was incorporated into the synthesis were determined. Results A total of 91 SRs met the inclusion criteria and were included in the final analysis. Over half of the SRs (85, 64.8%) used standard tools, such as the Jadad quality score and the Cochrane RoB tool, followed by adapted tools (n=23, 25.3%). Of the 85 SRs that assessed RoB, 29 (34.1%) presented domain-level assessment and 71 SRs (83.5%) included ranking of the studies based on RoB assessment. Of these 71, 35 (49.4%) used a cut-off threshold score and 26 (36.6%) required all criteria sum-up. Of the 85 SRs that assessed RoB, 48 (56.5%) incorporated RoB appraisal into the data synthesis. Conclusions Although most SRs of acupuncture for pain relief conducted some form of RoB assessment, nearly half of them failed to incorporate the RoB assessment into the synthesis.

scholarly journals Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on the Treatment of Acne Vulgaris: Cross-Sectional Analysis

10.2196/16978 ◽  
2020 ◽  
Vol 3 (1) ◽  
pp. e16978 ◽  
Author(s):  
Ryan Ottwell ◽  
Taylor C Rogers ◽  
J Michael Anderson ◽  
Austin Johnson ◽  
Matt Vassar
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
Acne Vulgaris ◽  
Cross Sectional Study ◽  
Primary Objective ◽  
Risk Of Bias ◽  
Selective Reporting ◽  
Cross Sectional ◽  
Bias Assessment ◽  
Meta Analyses

Background Spin is the misrepresentation of study findings, which may positively or negatively influence the reader’s interpretation of the results. Little is known regarding the prevalence of spin in abstracts of systematic reviews, specifically systematic reviews pertaining to the management and treatment of acne vulgaris. Objective The primary objective of this study was to characterize and determine the frequency of the most severe forms of spin in systematic review abstracts and to evaluate whether various study characteristics were associated with spin. Methods Using a cross-sectional study design, we searched PubMed and EMBASE for systematic reviews focusing on the management and treatment of acne vulgaris. Our search returned 316 studies, of which 36 were included in our final sample. To be included, each systematic review must have addressed either pharmacologic or nonpharmacologic treatment of acne vulgaris. These studies were screened, and data were extracted in duplicate by two blinded investigators. We analyzed systematic review abstracts for the nine most severe types of spin. Results Spin was present in 31% (11/36) of abstracts. A total of 12 examples of spin were identified in the 11 abstracts containing spin, with one abstract containing two instances of spin. The most common type of spin, selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention, was identified five times (5/12, 42%). A total of 44% (16/36) of studies did not report a risk of bias assessment. Of the 11 abstracts containing spin, six abstracts (55%) had not reported a risk of bias assessment or performed a risk of bias assessment but did not discuss it. Spin in abstracts was not significantly associated with a specific intervention type, funding source, or journal impact factor. Conclusions Spin is present in the abstracts of systematic reviews and meta-analyses covering the treatment of acne vulgaris. This paper raises awareness of spin in abstracts and emphasizes the importance of its recognition, which may lead to fewer incidences of spin in future studies.

scholarly journals Risk of bias in Cochrane systematic reviews: assessments of risk related to attrition bias are highly inconsistent

2018 ◽  
Author(s):  
Andrija Babic ◽  
Ruzica Tokalic ◽  
João Amílcar Silva Cunha ◽  
Ivana Novak ◽  
Jelena Suto ◽  
...  
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
Risk Of Bias ◽  
Systematic Review Methodology ◽  
Attrition Bias ◽  
Included Trial ◽  
Cochrane Reviews ◽  
Cochrane Database ◽  
Golden Standard ◽  
Improve Reliability

AbstractBackgroundAn important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews (CSRs) are considered golden standard regarding systematic review methodology, but Cochrane’s instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors’ assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in CSRs of interventions published in the Cochrane Database of Systematic Reviews (CDSR).MethodsWe analyzed CSRs published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many CSRs had different judgments for the same supporting explanations.ResultsIn the main analysis we included 10292 judgments and supporting explanations for attrition bias from 729 CSRs. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of CSR authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same CSR.ConclusionWe found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane’s risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.

scholarly journals Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on the Treatment of Acne Vulgaris: Cross-Sectional Analysis (Preprint)

2019 ◽  
Author(s):  
Ryan Ottwell ◽  
Taylor C Rogers ◽  
J Michael Anderson ◽  
Austin Johnson ◽  
Matt Vassar
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
Acne Vulgaris ◽  
Cross Sectional Study ◽  
Primary Objective ◽  
Risk Of Bias ◽  
Selective Reporting ◽  
Cross Sectional ◽  
Bias Assessment ◽  
Meta Analyses

BACKGROUND Spin is the misrepresentation of study findings, which may positively or negatively influence the reader’s interpretation of the results. Little is known regarding the prevalence of spin in abstracts of systematic reviews, specifically systematic reviews pertaining to the management and treatment of acne vulgaris. OBJECTIVE The primary objective of this study was to characterize and determine the frequency of the most severe forms of spin in systematic review abstracts and to evaluate whether various study characteristics were associated with spin. METHODS Using a cross-sectional study design, we searched PubMed and EMBASE for systematic reviews focusing on the management and treatment of acne vulgaris. Our search returned 316 studies, of which 36 were included in our final sample. To be included, each systematic review must have addressed either pharmacologic or nonpharmacologic treatment of acne vulgaris. These studies were screened, and data were extracted in duplicate by two blinded investigators. We analyzed systematic review abstracts for the nine most severe types of spin. RESULTS Spin was present in 31% (11/36) of abstracts. A total of 12 examples of spin were identified in the 11 abstracts containing spin, with one abstract containing two instances of spin. The most common type of spin, <i>selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention,</i> was identified five times (5/12, 42%). A total of 44% (16/36) of studies did not report a risk of bias assessment. Of the 11 abstracts containing spin, six abstracts (55%) had not reported a risk of bias assessment or performed a risk of bias assessment but did not discuss it. Spin in abstracts was not significantly associated with a specific intervention type, funding source, or journal impact factor. CONCLUSIONS Spin is present in the abstracts of systematic reviews and meta-analyses covering the treatment of acne vulgaris. This paper raises awareness of spin in abstracts and emphasizes the importance of its recognition, which may lead to fewer incidences of spin in future studies.

A Critical Overview of Systematic Reviews of Chemotherapy for Advanced and Locally Advanced Pancreatic Cancer using both AMSTAR2 and ROBIS as Quality Assessment Tools

2020 ◽  
Vol 15 ◽  
Author(s):  
Amit Dang ◽  
Surendar Chidirala ◽  
Prashanth Veeranki ◽  
BN Vallish
Keyword(s):  
Pancreatic Cancer ◽  
High Risk ◽  
Systematic Reviews ◽  
Locally Advanced ◽  
Advanced Pancreatic Cancer ◽  
Risk Of Bias ◽  
Low Risk ◽  
Locally Advanced Pancreatic Cancer ◽  
Grade 3 ◽  
Critical Overview

Background: We performed a critical overview of published systematic reviews (SRs) of chemotherapy for advanced and locally advanced pancreatic cancer, and evaluated their quality using AMSTAR2 and ROBIS tools. Materials and Methods: PubMed and Cochrane Central Library were searched for SRs on 13th June 2020. SRs with metaanalysis which included only randomized controlled trials and that had assessed chemotherapy as one of the treatment arms were included. The outcome measures, which were looked into, were progression-free survival (PFS), overall survival (OS), and adverse events (AEs) of grade 3 or above. Two reviewers independently assessed all the SRs with both ROBIS and AMSTAR2. Results: Out of the 1,879 identified records, 26 SRs were included for the overview. Most SRs had concluded that gemcitabine-based combination regimes, prolonged OS and PFS, but increased the incidence of grade 3-4 toxicities, when compared to gemcitabine monotherapy, but survival benefits were not consistent when gemcitabine was combined with molecular targeted agents. As per ROBIS, 24/26 SRs had high risk of bias, with only 1/26 SR having low risk of bias. As per AMSTAR2, 25/26 SRs had critically low, and 1/26 SR had low, confidence in the results. The study which scored ‘low’ risk of bias in ROBIS scored ‘low confidence in results’ in AMSTAR2. The inter-rater reliability for scoring the overall confidence in the SRs with AMSTAR2 and the overall domain in ROBIS was substantial; ROBIS: kappa=0.785, SEM=0.207, p<0.001; AMSTAR2: kappa=0.649, SEM=0.323, p<0.001. Conclusion: Gemcitabine-based combination regimens can prolong OS and PFS but also worsen AEs when compared to gemcitabine monotherapy. The included SRs have an overall low methodological quality and high risk of bias as per AMSTAR2 and ROBIS respectively.

Remotely Delivered Monitoring and Management of Diabetes-Related Foot Disease: An Overview of Systematic Reviews

2021 ◽  
pp. 193229682110124
Author(s):  
Aaron Drovandi ◽  
Shannon Wong ◽  
Leonard Seng ◽  
Benjamin Crowley ◽  
Chanika Alahakoon ◽  
...  
Keyword(s):  
Cost Effectiveness ◽  
Systematic Reviews ◽  
Urban Areas ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Limb Amputation ◽  
Current Evidence ◽  
Or Management ◽  
Foot Disease ◽  
Poor Reporting

Background: Diabetes-related foot disease (DFD) management requires input from multiple healthcare professionals, and has worse outcomes for people living in remote localities by comparison to urban areas. Remotely delivered healthcare may reduce this disparity. This overview summarizes current evidence on the effectiveness, stakeholder perceptions, and cost-effectiveness of remotely delivered healthcare for DFD. Methods: A search of 5 databases was conducted to identify systematic reviews published between January 2000 and June 2020. Eligible reviews were those evaluating remotely delivered monitoring or management of patients at risk of or with active DFD, or clinicians managing these patients. Risk of bias was assessed using the AMSTAR-2 tool. Results: Eight reviews were eligible for inclusion, including 88 primary studies and 8509 participants, of which 36 studies involving 4357 participants evaluated remotely delivered monitoring or management of DFD. Only one review had a low risk of bias, with most reviews demonstrating limited search strategies and poor reporting of participants. Evidence on effectiveness was mixed, with meta-analyses demonstrating long-term ulcer healing and mortality were not significantly different between telehealth and standard care groups, although the lower-limb amputation rate was significantly decreased in one meta-analysis. Perceptions of telehealth by patients and clinicians were generally positive, whilst acknowledging limitations relating to access and use. Cost-effectiveness data were limited, with poor reporting preventing clear conclusions. Conclusions: Remotely delivered healthcare of DFD is well received by patients and clinicians, but its effectiveness is unclear. High quality trials are needed to evaluate the risks and benefits of remotely delivered DFD management.

scholarly journals Concordance between systematic reviews of randomized controlled trials in assisted reproduction: an overview

2020 ◽  
Vol 2020 (4) ◽  
Author(s):  
Mariano Mascarenhas ◽  
Theodoros Kalampokas ◽  
Sesh Kamal Sunkara ◽  
Mohan S Kamath
Keyword(s):  
Assisted Reproduction ◽  
Systematic Reviews ◽  
Selective Reporting ◽  
Methodological Issues ◽  
Quality Of Evidence ◽  
Randomized Controlled ◽  
Cochrane Reviews ◽  
Secondary Outcomes ◽  
Meta Analyses

Abstract STUDY QUESTION Are systematic reviews published within a 3-year period on interventions in ART concordant in their conclusions? SUMMARY ANSWER The majority of the systematic reviews published within a 3-year period in the field of assisted reproduction on the same topic had discordant conclusions. WHAT IS KNOWN ALREADY Systematic reviews and meta-analyses have now replaced individual randomized controlled trials (RCTs) at the top of the evidence pyramid. There has been a proliferation of systematic reviews and meta-analyses, many of which suffer from methodological issues and provide varying conclusions. STUDY DESIGN, SIZE, DURATION We assessed nine interventions in women undergoing ART with at least three systematic reviews each, published from January 2015 to December 2017. PARTICIPANTS/MATERIALS, SETTING, METHODS The systematic reviews which included RCTs were considered eligible for inclusion. The primary outcome was extent of concordance between systematic reviews on the same topic. Secondary outcomes included assessment of quality of systematic reviews, differences in included studies in meta-analyses covering the same search period, selective reporting and reporting the quality of evidence. MAIN RESULTS AND THE ROLE OF CHANCE Concordant results and conclusions were found in only one topic, with reviews in the remaining eight topics displaying partial discordance. The AMSTAR grading for the majority of the non-Cochrane reviews was critically low whilst it was categorized as high for all of the Cochrane reviews. For three of the nine topics, none of the included systematic reviews assessed the quality of evidence. We were unable to assess selective reporting as most of the reviews did not have a pre-specified published protocol. LIMITATIONS, REASONS FOR CAUTION We were limited by the high proportion of reviews lacking a pre-specified protocol, which made it impossible to assess for selective reporting. Furthermore, many reviews did not specify primary and secondary outcomes which made it difficult to assess reporting bias. All the authors of this review were Cochrane review authors which may introduce some assessment bias. The categorization of the review’s conclusions as beneficial, harmful or neutral was subjective, depending on the tone and wording of the conclusion section of the review. WIDER IMPLICATIONS OF THE FINDINGS The majority of the systematic reviews published within a 3-year period on the same topic in the field of assisted reproduction revealed discordant conclusions and suffered from serious methodological issues, hindering the process of informed healthcare decision-making. STUDY FUNDING/COMPETING INTEREST(S) All the authors are Cochrane authors. M.S.K. is an editorial board member of Cochrane Gynaecology and Fertility group. No grant from funding agencies in the public, commercial or not-for-profit sectors was obtained.

Sign in / Sign up

Export Citation Format

Share Document