Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol

Abstract Background Systematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies. Methods An electronic search will be conducted using PubMed, Embase, ACM Digital Library, and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, and the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed. Discussion Automating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis and help to inspire further studies in this field. Systematic review registration PROSPERO CRD42020170815 (28 April 2020).

Download Full-text

A Research Protocol for a Systematic Review of Automatic Literature Screening in Medical Evidence Synthesis

10.21203/rs.3.rs-62316/v1 ◽

2020 ◽

Author(s):

Yuelun Zhang ◽

Siyu Liang ◽

Yunying Feng ◽

Qing Wang ◽

Feng Sun ◽

...

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Data Extraction ◽

Evidence Synthesis ◽

Evidence Based ◽

Medical Evidence ◽

Registration Number ◽

Evidence Collection ◽

Automatic Methods ◽

And Performance

Abstract Background: Systematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies.Methods: An electronic search will be conducted using PubMed, Embase and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed.Discussion: Automating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis, and help to inspire further studies in this field. Registration: PROSPERO registration number CRD42020170815 (28 April 2020).

Download Full-text

The development and evaluation of an online application to assist in the extraction of data from graphs for use in systematic reviews

Wellcome Open Research ◽

10.12688/wellcomeopenres.14738.2 ◽

2019 ◽

Vol 3 ◽

pp. 157

Author(s):

Fala Cramond ◽

Alison O'Mara-Eves ◽

Lee Doran-Constant ◽

Andrew SC Rice ◽

Malcolm Macleod ◽

...

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Statistical Data ◽

Data Extraction ◽

Evidence Synthesis ◽

Online Application ◽

Web Based ◽

Graphical Data ◽

Data Extraction Tool ◽

Increase In Accuracy

Background: The extraction of data from the reports of primary studies, on which the results of systematic reviews depend, needs to be carried out accurately. To aid reliability, it is recommended that two researchers carry out data extraction independently. The extraction of statistical data from graphs in PDF files is particularly challenging, as the process is usually completely manual, and reviewers need sometimes to revert to holding a ruler against the page to read off values: an inherently time-consuming and error-prone process. Methods: To mitigate some of the above problems we integrated and customised two existing JavaScript libraries to create a new web-based graphical data extraction tool to assist reviewers in extracting data from graphs. This tool aims to facilitate more accurate and timely data extraction through a user interface which can be used to extract data through mouse clicks. We carried out a non-inferiority evaluation to examine its performance in comparison to standard practice. Results: We found that the customised graphical data extraction tool is not inferior to users’ prior preferred current approaches. Our study was not designed to show superiority, but suggests that there may be a saving in time of around 6 minutes per graph, accompanied by a substantial increase in accuracy. Conclusions: Our study suggests that the incorporation of this type of tool in online systematic review software would be beneficial in facilitating the production of accurate and timely evidence synthesis to improve decision-making.

Download Full-text

Promoting evidence-based practice: training health professionals for the evidence synthesis

Revista da Escola de Enfermagem da USP ◽

10.1590/1980-220x-reeusp-2021-0180 ◽

2021 ◽

Vol 55 ◽

Author(s):

Daniela Filipa Batista Cardoso ◽

Diana Gabriela Simões Marques Santos ◽

Joana Filipa Cunha Rodrigues ◽

Nichole Bento ◽

Rogério Manuel Clemente Rodrigues ◽

...

Keyword(s):

Systematic Review ◽

Training Program ◽

Systematic Reviews ◽

Health Professionals ◽

Evidence Based Practice ◽

Evidence Synthesis ◽

Evidence Based ◽

Centre Of Excellence ◽

Satisfaction Evaluation ◽

Students Satisfaction

ABSTRACT Objective: To report the experience of the Portugal Centre For Evidence Based Practice (PCEBP): a JBI Centre of Excellence in the training of health professionals, researchers, and professors in the Comprehensive Systematic Review Training Program, a course on Evidence Synthesis, specifically on Systematic Literature Reviews. Method: This article aims to report the experience of the Portugal Centre For Evidence Based Practice: a JBI Centre of Excellence in the implementation of the Comprehensive Systematic Review Training Program that trains health professionals, researchers, and teachers to develop Systematic Reviews, according to the JBI approach. Results: By the end of 2020, 11 editions of the course had been developed with 136 participants from different educational and health institutions, from different countries. As a result of the training of these participants, 13 systematic reviews were published in JBI Evidence Synthesis and 10 reviews were published in other journals. Conclusion: The reported results and the students’ satisfaction evaluation allow us to emphasize the relevance of the course for health professionals training on evidence synthesis.

Download Full-text

Evaluating methodological quality of Prognostic models Including Patient-reported HeAlth outcomes iN oncologY (EPIPHANY): a systematic review protocol

BMJ Open ◽

10.1136/bmjopen-2018-025054 ◽

2018 ◽

Vol 8 (10) ◽

pp. e025054 ◽

Cited By ~ 2

Author(s):

Nina Deliu ◽

Francesco Cottone ◽

Gary S Collins ◽

Amélie Anota ◽

Fabio Efficace

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Methodological Quality ◽

Data Extraction ◽

Prognostic Models ◽

Published Data ◽

Evidence Based ◽

Patient Reported ◽

Data Extraction Form

IntroductionWhile there is mounting evidence of the independent prognostic value of patient-reported outcomes (PROs) for overall survival (OS) in patients with cancer, it is known that the conduct of these studies may hold a number of methodological challenges. The aim of this systematic review is to evaluate the quality of published studies in this research area, in order to identify methodological and statistical issues deserving special attention and to also possibly provide evidence-based recommendations.Methods and analysisAn electronic search strategy will be performed in PubMed to identify studies developing or validating a prognostic model which includes PROs as predictors. Two reviewers will independently be involved in data collection using a predefined and standardised data extraction form including information related to study characteristics, PROs measures used and multivariable prognostic models. Studies selection will be reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, with data extraction form using fields from the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist for multivariable models. Methodological quality assessment will also be performed and will be based on prespecified domains of the CHARMS checklist. As a substantial heterogeneity of included studies is expected, a narrative evidence synthesis will also be provided.Ethics and disseminationGiven that this systematic review will use only published data, ethical permissions will not be required. Findings from this review will be published in peer-reviewed scientific journals and presented at major international conferences. We anticipate that this review will contribute to identify key areas of improvement for conducting and reporting prognostic factor analyses with PROs in oncology and will lay the groundwork for developing future evidence-based recommendations in this area of research.Prospero registration numberCRD42018099160.

Download Full-text

The development and evaluation of an online application to assist in the extraction of data from graphs for use in systematic reviews

Wellcome Open Research ◽

10.12688/wellcomeopenres.14738.1 ◽

2018 ◽

Vol 3 ◽

pp. 157

Author(s):

Fala Cramond ◽

Alison O'Mara-Eves ◽

Lee Doran-Constant ◽

Andrew SC Rice ◽

Malcolm Macleod ◽

...

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Statistical Data ◽

Data Extraction ◽

Evidence Synthesis ◽

Online Application ◽

Web Based ◽

Graphical Data ◽

Data Extraction Tool ◽

Increase In Accuracy

Background: The extraction of data from the reports of primary studies, on which the results of systematic reviews depend, needs to be carried out accurately. To aid reliability, it is recommended that two researchers carry out data extraction independently. The extraction of statistical data from graphs in PDF files is particularly challenging, as the process is usually completely manual, and reviewers need sometimes to revert to holding a ruler against the page to read off values: an inherently time-consuming and error-prone process. Methods: To mitigate some of the above problems we developed a new web-based graphical data extraction tool to assist reviewers in extracting data from graphs. This tool aims to facilitate more accurate and timely data extraction through a user interface which can be used to extract data through mouse clicks. We carried out a non-inferiority evaluation to examine its performance in comparison to standard practice. Results: We found that our new graphical data extraction tool is not inferior to users’ prior preferred current approaches. Our study was not designed to show superiority, but suggests that there may be a saving in time of around 6 minutes per graph, accompanied by a substantial increase in accuracy. Conclusions: Our study suggests that the incorporation of this type of tool in online systematic review software would be beneficial in facilitating the production of accurate and timely evidence synthesis to improve decision-making.

Download Full-text

Acting in the Face of Evidentiary Ambiguity, Bias, and Absence Arising from Systematic Reviews in Applied Environmental Science

10.32942/osf.io/4ertu ◽

2021 ◽

Author(s):

Trina Rytwinski ◽

Steven J Cooke ◽

Jessica J Taylor ◽

Dominique Roche ◽

Paul A Smith ◽

...

Keyword(s):

Systematic Review ◽

Decision Making ◽

Systematic Reviews ◽

Environmental Science ◽

Evidence Synthesis ◽

Empirical Studies ◽

Synthesis Method ◽

Evidence Base ◽

Evidence Based ◽

Evidence Based Decision Making

Evidence-based decision-making often depends on some form of a synthesis of previous findings. There is growing recognition that systematic reviews, which incorporate a critical appraisal of evidence, are the gold standard synthesis method in applied environmental science. Yet, on a daily basis, environmental practitioners and decision-makers are forced to act even if the evidence base to guide them is insufficient. For example, it is not uncommon for a systematic review to conclude that an evidence base is large but of low reliability. There are also instances where the evidence base is sparse (e.g., one or two empirical studies on a particular taxa or intervention), and no additional evidence arises from a systematic review. In some cases, the systematic review highlights considerable variability in the outcomes of primary studies, which in turn generates ambiguity (e.g., potentially context specific). When the environmental evidence base is ambiguous, biased, or lacking of new information, practitioners must still make management decisions. Waiting for new, higher validity research to be conducted is often unrealistic as many decisions are urgent. Here, we identify the circumstances that can lead to ambiguity, bias, and the absence of additional evidence arising from systematic reviews and provide practical guidance to resolve or handle these scenarios when encountered. Our perspective attempts to highlight that, with evidence synthesis, there may be a need to balance the spirit of evidence-based decision-making and the practical reality that management and conservation decisions and action is often time sensitive.

Download Full-text

New meniscal tears after ACL injury: what is the risk? A systematic review protocol

British Journal of Sports Medicine ◽

10.1136/bjsports-2017-097728 ◽

2017 ◽

Vol 52 (6) ◽

pp. 386-386 ◽

Cited By ~ 10

Author(s):

Guri Ranum Ekås ◽

Clare Ardern ◽

Hege Grindem ◽

Lars Engebretsen

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Data Extraction ◽

Evidence Synthesis ◽

Meta Analysis ◽

Demographic Data ◽

Acl Injury ◽

Risk Of Bias ◽

Meniscal Tears ◽

Best Evidence Synthesis

BackgroundSecondary meniscal tears after ACL injuries increase the risk of knee osteoarthritis. The current literature on secondary meniscal injuries after ACL injury is not consistent and may have methodological shortcomings. This protocol describes the methods of a systematic review investigating the rate of secondary meniscal injuries in children and adults after treatment (operative or non-operative) for ACL injury.MethodsWe will search electronic databases (Embase, Ovid Medline, Cochrane, CINAHL (Cumulative Index to Nursing and Allied Health Literature), SPORTDiscus, PEDro and Google Scholar) from database inception. Extracted data will include demographic data, methodology, intervention details and patient outcomes. Risk of bias will be assessed using the Newcastle Ottawa checklist for cohort studies. Article screening, eligibility assessment, risk of bias assessment and data extraction will be performed in duplicate by independent reviewers. A proportion meta-analysis will be performed if studies are homogeneous (I2<75%). If meta-analysis is precluded, data will be synthesised descriptively using best-evidence synthesis. The strength of recommendations and quality of evidence will be assessed using the Grading of Recommendations Assessment Development and Evaluation working group methodology.Ethics and disseminationThis protocol is written according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses, and was registered in the International Prospective Register of Systematic Reviews on 22 March 2016.Trial registration numberCRD42016036788.

Download Full-text

Methodology for Evidence Synthesis and Development of Comprehensive Evidence-Based Guidelines for Interventional Techniques in Chronic Spinal Pain

January 2018 - Pain Physician ◽

10.36076/ppj.2021.24.s1-s26 ◽

2021 ◽

Vol S1;24 (1;S1) ◽

pp. S1-S26

Keyword(s):

Systematic Review ◽

Clinical Practice ◽

Systematic Reviews ◽

Clinical Practice Guidelines ◽

Practice Guidelines ◽

Observational Studies ◽

Evidence Synthesis ◽

Evidence Based ◽

Interventional Techniques

BACKGROUND: The re-engineered definition of clinical guidelines in 2011 from the IOM (Institute of Medicine) states, “clinical practice guidelines are statements that include recommendations intended to optimize patient care that is informed by a systematic review of evidence and an assessment of the benefit and harms of alternative care options.” The revised definition distinguishes between the term “clinical practice guideline” and other forms of clinical guidance derived from widely disparate development processes, such as consensus statements, expert advice, and appropriate use criteria. OBJECTIVE: To assess the literature and develop methodology for evidence synthesis and development of comprehensive evidence-based guidelines for interventional techniques in chronic spinal pain. METHODS: A systematic review of the literature including methodology of guideline development encompassing GRADE approach for guidance on evidence synthesis with recommendations. RESULTS: Some of the many factors described in 2011 continue as of 2020 and impede the development of clinical practice guidelines. These impediments include biases due to a variety of conflicts and confluence of interest, inappropriate and poor methodological quality, poor writing and ambiguous presentation, projecting a view that these are not applicable to individual patients or too restrictive with the elimination of clinician autonomy, and overzealous and inappropriate recommendations, either positive, negative, or non-committal. Thus, ideally, a knowledgeable, multidisciplinary panel of experts with true lack of bias and confluence of interest must develop guidelines based on a systematic review of the existing evidence. This manuscript describes evidence synthesis from observational studies, various types of randomized controlled trials (RCTs), and, finally, methodological and reporting quality of systematic reviews. The manuscript also describes various methods utilized in the assessment of the quality of observational studies, diagnostic accuracy studies, RCTs, and systematic reviews. LIMITATIONS: Paucity of publications with appropriate evidence synthesis methodology in reference to interventional techniques. CONCLUSION: This review described comprehensive evidence synthesis derived from systematic reviews, including methodologic quality and bias measurement. The manuscript described various methods utilized in the assessment of the quality of the systematic reviews, RCTs, diagnostic accuracy studies, and observational studies. KEY WORDS: Evidence-based medicine (EBM), interventional pain management, evidence synthesis, methodological quality assessment, conflict of interest, confluence of interest, comparative effectiveness research (CER), clinical practice guidelines, systematic reviews, meta-analysis

Download Full-text

Brief Psychotherapy for Depression: A Systematic Review and Meta-Analysis

The International Journal of Psychiatry in Medicine ◽

10.2190/pm.43.2.c ◽

2012 ◽

Vol 43 (2) ◽

pp. 129-151 ◽

Cited By ~ 49

Author(s):

Jason A. Nieuwsma ◽

Ranak B. Trivedi ◽

Jennifer McDuffie ◽

Ian Kronish ◽

Dinesh Benjamin ◽

...

Keyword(s):

Systematic Review ◽

Problem Solving ◽

Cognitive Behavioral Therapy ◽

Systematic Reviews ◽

Behavioral Therapy ◽

Meta Analysis ◽

Cognitive Behavioral ◽

Evidence Based ◽

Brief Psychotherapy ◽

Problem Solving Therapy

Objective: Because evidence-based psychotherapies of 12 to 20 sessions can be perceived as too lengthy and time intensive for the treatment of depression in primary care, a number of studies have examined abbreviated psychotherapy protocols. The purpose of this study was to conduct a systematic review and meta-analysis to determine the efficacy of brief psychotherapy (i.e., < 8 sessions) for depression. Methods: We used combined literature searches in PubMed, EMBASE, PsycINFO, and an Internet-accessible database of clinical trials of psychotherapy to conduct two systematic searches: one for existing systematic reviews and another for randomized controlled trials (RCTs). Included studies examined evidence-based psychotherapy(s) of eight or fewer sessions, focused on adults with depression, contained an acceptable control condition, were published in English, and used validated measures of depressive symptoms. Results: We retained 2 systematic reviews and 15 RCTs evaluating cognitive behavioral therapy, problem-solving therapy, and mindfulness-based cognitive therapy. The systematic reviews found brief psychotherapies to be more efficacious than control, with effect sizes ranging from −0.33 to −0.25. Our meta-analysis found six to eight sessions of cognitive behavioral therapy to be more efficacious than control (ES −0.42, 95% CI −0.74 to −0.10, I2 = 56%). A sensitivity analysis controlled for statistical heterogeneity but showed smaller treatment effects (ES −0.24, 95% CI −0.42 to −0.06, I2 = 0%). Conclusions: Depression can be efficaciously treated with six to eight sessions of psychotherapy, particularly cognitive behavioral therapy and problem-solving therapy. Access to non-pharmacologic treatments for depression could be improved by training healthcare providers to deliver brief psychotherapies.

Download Full-text

Development of the ACTIVE framework to describe stakeholder involvement in systematic reviews

Journal of Health Services Research & Policy ◽

10.1177/1355819619841647 ◽

2019 ◽

Vol 24 (4) ◽

pp. 245-255 ◽

Cited By ~ 5

Author(s):

Alex Pollock ◽

Pauline Campbell ◽

Caroline Struthers ◽

Anneliese Synnot ◽

Jack Nunn ◽

...

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Health Professionals ◽

Public Involvement ◽

Review Process ◽

Data Extraction ◽

Stakeholder Involvement ◽

Decision Makers ◽

Literature Searching ◽

Health Decision

Objectives Involvement of patients, health professionals, and the wider public (‘stakeholders’) is seen to be beneficial to the quality, relevance and impact of research and may enhance the usefulness and uptake of systematic reviews. However, there is a lack of evidence and resources to guide researchers in how to actively involve stakeholders in systematic reviews. In this paper, we report the development of the ACTIVE framework to describe how stakeholders are involved in systematic reviews. Methods We developed a framework using methods previously described in the development of conceptual frameworks relating to other areas of public involvement, including: literature searching, data extraction, analysis, and categorization. A draft ACTIVE framework was developed and then refined after presentation at a conference workshop, before being applied to a subset of 32 systematic reviews. Data extracted from these systematic reviews, identified in a systematic scoping review, were categorized against pre-defined constructs, including: who was involved, how stakeholders were recruited, the mode of involvement, at what stage there was involvement and the level of control or influence. Results The final ACTIVE framework described whether patients, carers and/or families, and/or other stakeholders (including health professionals, health decision makers and funders) were involved. We defined: recruitment as either open or closed; the approach to involvement as either one-time, continuous or combined; and the method of involvement as either direct or indirect. The stage of involvement in reviews was defined using the Cochrane Ecosystem stages of a review. The level of control or influence was defined according to the roles and activities of stakeholders in the review process, and described as the ACTIVE continuum of involvement. Conclusions The ACTIVE framework provides a structure with which to describe key components of stakeholder involvement within a systematic review, and we have used this to summarize how stakeholders have been involved in a subset of varied systematic reviews. The ACTIVE continuum of involvement provides a new model that uses tasks and roles to detail the level of stakeholder involvement. This work has contributed to the development of learning resources aimed at supporting systematic review authors and editors to involve stakeholders in their systematic reviews. The ACTIVE framework may support the decision-making of systematic review authors in planning how to involve stakeholders in future reviews.

Download Full-text