scholarly journals RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials

2015 ◽  
Vol 23 (1) ◽  
pp. 193-201 ◽  
Author(s):  
Iain J Marshall ◽  
Joël Kuiper ◽  
Byron C Wallace
Keyword(s):  
Machine Learning ◽  
Clinical Trials ◽  
Systematic Reviews ◽  
Risk Of Bias ◽  
Reasonable Accuracy ◽  
Bias Assessment ◽  
Cochrane Database ◽  
The Difference ◽  
Using Data ◽  
Unclear Risk

Abstract Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments. Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR. Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v 56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR). Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses.

scholarly journals Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Ognjen Barcot ◽  
Matija Boric ◽  
Svjetlana Dosenovic ◽  
Marija Cavar ◽  
Antonia Jelicic Kadic ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Allocation Concealment ◽  
Risk Of Bias ◽  
Methodological Study ◽  
Bias Assessment ◽  
Surgical Trials ◽  
Surgical Interventions ◽  
Randomized Controlled ◽  
Cochrane Reviews ◽  
The Difference

Abstract Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

A critical appraisal of the risk of bias in systematic reviews and metaanalyses pertaining to COVID-19

Coronaviruses ◽  
2020 ◽  
Vol 01 ◽  
Author(s):  
Amit Dang ◽  
Sheshank Madiraju ◽  
Jagan Mohan Venkateswara Rao P ◽  
Navya Sri Gurram ◽  
Sandeep Digijarala ◽  
...  
Keyword(s):  
High Risk ◽  
Systematic Reviews ◽  
Risk Of Bias ◽  
Therapy Outcomes ◽  
Data Synthesis ◽  
Bias Assessment ◽  
Meta Analyses ◽  
Unclear Risk ◽  
Overall Risk

Background: We critically evaluated the risk of bias in published systematic reviews (SRs) and meta-analyses (MAs) pertaining to COVID-19 using ROBIS tool. Materials And Methods: Medline and Cochrane Central Library were searched for SRs/MAs on 14th May 2020, including studies of all designs describing various facets of COVID-19 in humans; no restrictions were applied for interventions, comparators, and outcomes. Two reviewers independently assessed all the SRs/MAs with ROBIS. Results: Out of 204 identified records, 48 SRs/MAs were included. The most frequently reviewed topics were therapy outcomes, diagnosis, and comorbidities (15, 8, and 6 papers respectively). Only 29/48(60.41%) papers had made a mention of using PRISMA or other guidelines for drafting the SR/MA. Only 5/48(10.42%) of all included SRs/MAs had low overall risk of bias as per ROBIS tool; 41/48(85.42%) had high risk of bias, 2/48(4.17%) had unclear risk of bias. The highest proportion of bias was found in data synthesis and findings (30/48, 62.50% of studies had high risk of bias), followed by study identification and selection (29/48, 60.42%). The IRR for methodological quality assessment was substantial, with the Cohen’s kappa values being 0.64, 0.68, 0.62, and 0.75 for domains 1-4 of ROBIS tool, and 0.66 for overall risk of bias assessment. Conclusion: There are serious concerns about the methodology employed to generate the results of the SRs/ MAs pertaining to COVID-19, with ‘quantity’ seemingly being given more importance than ‘quality’ of the paper.

scholarly journals Assessments of risk of bias in systematic reviews of observational nutritional epidemiologic studies are often not appropriate or comprehensive: a methodological study

2021 ◽  
pp. e000248
Author(s):  
Dena Zeraatkar ◽  
Alana Kohut ◽  
Arrti Bhasin ◽  
Rita E Morassut ◽  
Isabella Churchill ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Epidemiologic Studies ◽  
Risk Of Bias ◽  
Selective Reporting ◽  
Methodological Study ◽  
Essential Component ◽  
Cochrane Database

BackgroundAn essential component of systematic reviews is the assessment of risk of bias. To date, there has been no investigation of how reviews of non-randomised studies of nutritional exposures (called ‘nutritional epidemiologic studies’) assess risk of bias.ObjectiveTo describe methods for the assessment of risk of bias in reviews of nutritional epidemiologic studies.MethodsWe searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews (Jan 2018–Aug 2019) and sampled 150 systematic reviews of nutritional epidemiologic studies.ResultsMost reviews (n=131/150; 87.3%) attempted to assess risk of bias. Commonly used tools neglected to address all important sources of bias, such as selective reporting (n=25/28; 89.3%), and frequently included constructs unrelated to risk of bias, such as reporting (n=14/28; 50.0%). Most reviews (n=66/101; 65.3%) did not incorporate risk of bias in the synthesis. While more than half of reviews considered biases due to confounding and misclassification of the exposure in their interpretation of findings, other biases, such as selective reporting, were rarely considered (n=1/150; 0.7%).ConclusionReviews of nutritional epidemiologic studies have important limitations in their assessment of risk of bias.

scholarly journals Effects of Composite Attachments on Orthodontic Clear Aligners Therapy: A Systematic Review

Materials ◽  
2022 ◽  
Vol 15 (2) ◽  
pp. 533
Author(s):  
Riccardo Nucera ◽  
Carolina Dolci ◽  
Angela Mirea Bellocchio ◽  
Stefania Costa ◽  
Serena Barbera ◽  
...  
Keyword(s):  
Systematic Review ◽  
Clinical Trials ◽  
Data Extraction ◽  
Risk Of Bias ◽  
Bias Assessment ◽  
Bodily Movement ◽  
Study Selection ◽  
Medium Risk ◽  
Clear Aligner ◽  
Anterior Root

This systematic review aims to highlight the differences between different clear aligner therapies that differ in the presence of attachments or in attachment configuration. Eight electronic databases were searched up to March 2020. Two authors independently proceeded to study selection, data extraction, and risk of bias assessment. The analysis of the results was carried out examining six groups of movements (mesio-distal tipping/bodily movement; anterior bucco-lingual tipping/root torque; posterior bucco-lingual tipping/expansion; intrusion; extrusion; rotation). Five clinical trials were selected and all of them showed a medium risk of bias. Literature showed that attachments mostly increase the effectiveness of orthodontic treatment with clear aligners, improving anterior root torque, rotation, and mesio-distal (M-D) movement; they are also important to increase posterior anchorage. However, some articles showed contradictory or not statistically significant results. Attachments also seem to improve intrusion, but the evidence about this movement, as well as extrusion, is lacking. No studies evaluated posterior bucco-lingual tipping/expansion. Further clinical trials are strongly suggested to clarify the influence of attachments and their number, size, shape, and position on each orthodontic movement.

scholarly journals The Use of Green Coffee Extract as a Weight Loss Supplement: A Systematic Review and Meta-Analysis of Randomised Clinical Trials

2011 ◽  
Vol 2011 ◽  
pp. 1-6 ◽  
Author(s):  
Igho Onakpoya ◽  
Rohini Terry ◽  
Edzard Ernst
Keyword(s):  
Weight Loss ◽  
Clinical Trials ◽  
Methodological Quality ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Significant Heterogeneity ◽  
Green Coffee ◽  
Significant Difference ◽  
Using Data

The purpose of this paper is to assess the efficacy of green coffee extract (GCE) as a weight loss supplement, using data from human clinical trials. Electronic and nonelectronic searches were conducted to identify relevant articles, with no restrictions in time or language. Two independent reviewers extracted the data and assessed the methodological quality of included studies. Five eligible trials were identified, and three of these were included. All studies were associated with a high risk of bias. The meta-analytic result reveals a significant difference in body weight in GCE compared with placebo (mean difference: kg; 95%CI: , ). The magnitude of the effect is moderate, and there is significant heterogeneity amongst the studies. It is concluded that the results from these trials are promising, but the studies are all of poor methodological quality. More rigorous trials are needed to assess the usefulness of GCE as a weight loss tool.

Quantifying changes in bicycle volumes using crowdsourced data

2022 ◽  
pp. 239980832110661
Author(s):  
Ali Al-Ramini ◽  
Mohammad A Takallou ◽  
Daniel P Piatkowski ◽  
Fadi Alsaleem
Keyword(s):  
Machine Learning ◽  
The United States ◽  
Crowdsourced Data ◽  
Machine Learning Approach ◽  
Bicycle Infrastructure ◽  
The Difference ◽  
Infrastructure Investments ◽  
Using Data ◽  
The Impact ◽  
The City

Most cities in the United States lack comprehensive or connected bicycle infrastructure; therefore, inexpensive and easy-to-implement solutions for connecting existing bicycle infrastructure are increasingly being employed. Signage is one of the promising solutions. However, the necessary data for evaluating its effect on cycling ridership is lacking. To overcome this challenge, this study tests the potential of using readily-available crowdsourced data in concert with machine-learning methods to provide insight into signage intervention effectiveness. We do this by assessing a natural experiment to identify the potential effects of adding or replacing signage within existing bicycle infrastructure in 2019 in the city of Omaha, Nebraska. Specifically, we first visually compare cycling traffic changes in 2019 to those from the previous two years (2017–2018) using data extracted from the Strava fitness app. Then, we use a new three-step machine-learning approach to quantify the impact of signage while controlling for weather, demographics, and street characteristics. The steps are as follows: Step 1 (modeling and validation) build and train a model from the available 2017 crowdsourced data (i.e., Strava, Census, and weather) that accurately predicts the cycling traffic data for any street within the study area in 2018; Step 2 (prediction) use the model from Step 1 to predict bicycle traffic in 2019 while assuming new signage was not added; Step 3 (impact evaluation) use the difference in prediction from actual traffic in 2019 as evidence of the likely impact of signage. While our work does not demonstrate causality, it does demonstrate an inexpensive method, using readily-available data, to identify changing trends in bicycling over the same time that new infrastructure investments are being added.

Auriculoterapia para o tratamento da ansiedade em estudantes universitários: revisão sistemática

2021 ◽  
Author(s):  
◽  
Lara dos Santos Silva
Keyword(s):  
Clinical Trials ◽  
Systematic Reviews ◽  
Ad Hoc ◽  
Cochrane Collaboration ◽  
Risk Of Bias ◽  
Meta Analyses ◽  
Tecnología Educativa

A ansiedade está entre os transtornos mentais de maior prevalência na população mundial, sendo a segunda causa de incapacidade mental no mundo. Dentre a população suscetível à ansiedade, destacam-se os estudantes universitários, em decorrência das situações vivenciadas no âmbito acadêmico. Os tratamentos da ansiedade podem estar pautados em recursos medicamentosos e não medicamentosos. Dentre os não medicamentosos, destacam-se as Práticas Integrativas e Complementares em Saúde e, dentre estas, a auriculoterapia. O objetivo geral deste estudo foi s intetizar as evidências disponíveis na literatura científica sobre o uso da auriculoterapia no tratamento da ansiedade em estudantes universitários. Trata-se de uma revisão sistemática da literatura, realizada em 16 bases de dados, com início em 20 de janeiro e revisão até 27 de março de 2021. Todo o processo de busca e seleção dos artigos foi feito de modo independente e pareado, e, quando necessário, um terceiro pesquisador com experiência na temática auxiliava como ad hoc na resolução dos conflitos. O relato da revisão pautou-se na diretriz Preferred Reporting Items for Systematic Reviews and Meta-Analyses, e o protocolo da revisão foi cadastrado na plataforma International Prospective Register of Systematic Reviews (registro No CRD42020205968). Utilizou-se as diretrizes Standards for Reporting Interventions in Clinical Trials of Acupuncture, Cochrane Collaboration Risk of Bias Tool, e escala de JADAD para avaliação dos artigos. A amostra final foi composta por cinco artigos. A análise dos mesmos mostrou que a auriculoterapia contribuiu para tratar a ansiedade de estudantes universitários. O uso desta intervenção deve ser considerado pelas instituições formadoras para promoção da saúde mental nesta população, e pode contribuir para a melhora do desempenho acadêmico, redução das taxas de evasão, e melhora da qualidade de vida aos universitários. O estudo também possibilitou a elaboração de uma tecnologia educativa no formato de cartilha, com o objetivo de esclarecer aos estudantes universitários sobre a ansiedade como um problema de saúde relevante na população, bem como apresentar a auriculoterapia como uma possibilidade terapêutica. Pretende-se que, em momento oportuno, a mesma seja divulgada em formato eletrônico para toda a comunidade acadêmica. Palavras-chave: Auriculoterapia. Terapia auricular. Ansiedade. Estudantes.

scholarly journals Risk of bias assessment of test comparisons was uncommon in comparative accuracy systematic reviews: an overview of reviews

2020 ◽  
Vol 127 ◽  
pp. 167-174 ◽  
Author(s):  
Bada Yang ◽  
Yasaman Vali ◽  
Anahita Dehmoobad Sharifabadi ◽  
Isobel Marion Harris ◽  
Sophie Beese ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Risk Of Bias ◽  
Bias Assessment ◽  
Overview Of Reviews ◽  
Comparative Accuracy
Sign in / Sign up

Export Citation Format

Share Document