The views of health guideline developers on the use of automation in health evidence synthesis

Abstract Background The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis – systematic reviews and health guidelines -- to be continually kept up-to-date maintain the most up-to-date data. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore their opinions on the potential use of automation are crucial. Methods The objective of this study was to analyse the attitudes of guideline developers towards the use of machine learning and crowd-sourcing in evidence. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Relative Advantage , Compatibility , Complexity , Trialability , and Observability . Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyse the data. First, transcripts were coded with a deductive approach using Rogers’ Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach. Results Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (ie. Compatibility in the Diffusion of Innovations framework. Participants were also concerned with Relative Advantage and Observability , which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in methodology of automation software. Participants were noticeably less interested in Complexity and Trialability , which were discussed infrequently. These results were reasonably consistent across all participants. Conclusions If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.

Download Full-text

The views of health guideline developers on the use of automation in health evidence synthesis

Systematic Reviews ◽

10.1186/s13643-020-01569-2 ◽

2021 ◽

Vol 10 (1) ◽

Author(s):

Anneliese Arno ◽

Julian Elliott ◽

Byron Wallace ◽

Tari Turner ◽

James Thomas

Keyword(s):

Diffusion Of Innovation ◽

Systematic Reviews ◽

Diffusion Of Innovations ◽

New Technologies ◽

Guideline Development ◽

Evidence Synthesis ◽

Analytical Framework ◽

Relative Advantage ◽

Full Potential ◽

Novel Technologies

Abstract Background The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis—systematic reviews and health guidelines—to be continually kept up to date. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore, their opinions on the potential use of automation are crucial. Methods The objective of this study was to analyze the attitudes of guideline developers towards the use of automation in health evidence synthesis. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Relative Advantage, Compatibility, Complexity, Trialability, and Observability. Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyze the data. First, transcripts were coded with a deductive approach using Rogers’ Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach. Results Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (i.e., Compatibility in the Diffusion of Innovations framework). Participants were also concerned with Relative Advantage and Observability, which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in the methodology of automation software. Participants were noticeably less interested in Complexity and Trialability, which were discussed infrequently. These results were reasonably consistent across all participants. Conclusions If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.

Download Full-text

The views of health guideline developers on the use of automation in health evidence synthesis

10.21203/rs.3.rs-23742/v2 ◽

2020 ◽

Author(s):

Anneliese Downey Arno ◽

Julian Elliott ◽

Byron Wallace ◽

Tari Turner ◽

James Thomas

Keyword(s):

Diffusion Of Innovation ◽

Systematic Reviews ◽

Diffusion Of Innovations ◽

New Technologies ◽

Guideline Development ◽

Evidence Synthesis ◽

Analytical Framework ◽

Relative Advantage ◽

Full Potential ◽

Novel Technologies

Abstract Background: The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis – systematic reviews and health guidelines -- to be continually kept up to date. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore their opinions on the potential use of automation are crucial. Methods: The objective of this study was to analyze the attitudes of guideline developers towards the use of automation in health evidence synthesis. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Relative Advantage, Compatibility, Complexity, Trialability, and Observability.Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyze the data. First, transcripts were coded with a deductive approach using Rogers’ Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach.Results: Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (i.e. Compatibility in the Diffusion of Innovations framework). Participants were also concerned with Relative Advantage and Observability, which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in methodology of automation software. Participants were noticeably less interested in Complexity and Trialability, which were discussed infrequently. These results were reasonably consistent across all participants. Conclusions: If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.

Download Full-text

Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods

BMJ Global Health ◽

10.1136/bmjgh-2018-000882 ◽

2019 ◽

Vol 4 (Suppl 1) ◽

pp. e000882 ◽

Cited By ~ 51

Author(s):

Kate Flemming ◽

Andrew Booth ◽

Ruth Garside ◽

Özge Tunçalp ◽

Jane Noyes

Keyword(s):

Qualitative Research ◽

Systematic Reviews ◽

Guideline Development ◽

Evidence Synthesis ◽

Complex Interventions ◽

Real Life ◽

Synthesis Methods ◽

Qualitative Evidence Synthesis ◽

Qualitative Evidence ◽

Broad Term

This paper is one of a series exploring the implications of complexity for systematic reviews and guideline development, commissioned by the WHO. The paper specifically explores the role of qualitative evidence synthesis. Qualitative evidence synthesis is the broad term for the group of methods used to undertake systematic reviews of qualitative research evidence. As an approach, qualitative evidence synthesis is increasingly recognised as having a key role to play in addressing questions relating to intervention or system complexity, and guideline development processes. This is due to the unique role qualitative research can play in establishing the relative importance of outcomes, the acceptability, fidelity and reach of interventions, their feasibility in different settings and potential consequences on equity across populations. This paper outlines the purpose of qualitative evidence synthesis, provides detail of how qualitative evidence syntheses can help establish understanding and explanation of the complexity that can occur in relation to both interventions and systems, and how qualitative evidence syntheses can contribute to evidence to decision frameworks. It provides guidance for the choice of qualitative evidence synthesis methods in the context of guideline development for complex interventions, giving ‘real life’ examples of where this has occurred. Information to support decision-making around choice qualitative evidence synthesis methods in the context of guideline development is provided. Approaches for reporting qualitative evidence syntheses are discussed alongside mechanisms for assessing confidence in the findings of a review.

Download Full-text

Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research

Wellcome Open Research ◽

10.12688/wellcomeopenres.17141.1 ◽

2021 ◽

Vol 6 ◽

pp. 210

Author(s):

Ian Shemilt ◽

Anneliese Arno ◽

James Thomas ◽

Theo Lorenc ◽

Claire Khouja ◽

...

Keyword(s):

Machine Learning ◽

Cost Effectiveness ◽

Active Learning ◽

Systematic Reviews ◽

Production Costs ◽

Sensitivity Analyses ◽

Relative Advantage ◽

Base Case ◽

Network Graph ◽

Microsoft Academic

Background: Conventionally, searching for eligible articles to include in systematic reviews and maps of research has relied primarily on information specialists conducting Boolean searches of multiple databases and manually processing the results, including deduplication between these multiple sources. Searching one, comprehensive source, rather than multiple databases, could save time and resources. Microsoft Academic Graph (MAG) is potentially such a source, containing a network graph structure which provides metadata that can be exploited in machine learning processes. Research is needed to establish the relative advantage of using MAG as a single source, compared with conventional searches of multiple databases. This study sought to establish whether: (a) MAG is sufficiently comprehensive to maintain our living map of coronavirus disease 2019 (COVID-19) research; and (b) eligible records can be identified with an acceptably high level of specificity. Methods: We conducted a pragmatic, eight-arm cost-effectiveness analysis (simulation study) to assess the costs, recall and precision of our semi-automated MAG-enabled workflow versus conventional searches of MEDLINE and Embase (with and without machine learning classifiers, active learning and/or fixed screening targets) for maintaining a living map of COVID-19 research. Resource use data (time use) were collected from information specialists and other researchers involved in map production. Results: MAG-enabled workflows dominated MEDLINE-Embase workflows in both the base case and sensitivity analyses. At one month (base case analysis) our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified n=469 more new, eligible articles for inclusion in our living map – and cost £3,179 GBP ($5,691 AUD) less – than conventional MEDLINE-Embase searches without any automation or fixed screening targets. Conclusions: MAG-enabled continuous surveillance workflows have potential to revolutionise study identification methods for living maps, specialised registers, databases of research studies and/or collections of systematic reviews, by increasing their recall and coverage, whilst reducing production costs.

Download Full-text

Testing An Adoption Decision-Making Model of Nursery and Greenhouse Growers’ Water Reuse in the United States

Water ◽

10.3390/w11122470 ◽

2019 ◽

Vol 11 (12) ◽

pp. 2470

Author(s):

Alexa Lamm ◽

Laura Warner ◽

Abraham Tidwell ◽

Kevan Lamm ◽

Paul Fisher ◽

...

Keyword(s):

Water Treatment ◽

Diffusion Of Innovations ◽

Water Reuse ◽

New Technologies ◽

The United States ◽

Relative Advantage ◽

Adoption Decision ◽

Treatment Technologies ◽

Greenhouse Growers ◽

Decision Making Model

There is an increasing necessity to implement water treatment technologies in order to optimize the use of freshwater resources as the global nursery and greenhouse industry grows. Unfortunately, their adoption has been limited. This study tested a conceptual model for technology adoption based on the Theory of Diffusion of Innovations in tandem with Adaption-Innovation Theory and Critical Thinking Style literature. Using a series of linear and logistic regressions, three characteristics of an innovation—relative advantage, complexity, and trialability—were identified as significant drivers of growers’ decisions to implement water treatment technologies. Growers who seek information when thinking critically and are more innovative when solving problems did not perceive new technologies to be compatible or to possess a relative advantage over their current systems. The results suggested most growers are unsure of how new technologies fit into their existing operations. Creating opportunities for growers to get hands on experience with new technologies, such as a field day, could assist in increasing growers’ implementation. In addition, developing a series of online videos highlighting how to use, adapt and troubleshoot the equipment would greatly enhance chances of long-term adoption.

Download Full-text

Towards a new model for producing evidence-based guidelines: a qualitative study of current approaches and opportunities for innovation among Australian guideline developers

F1000Research ◽

10.12688/f1000research.19661.1 ◽

2019 ◽

Vol 8 ◽

pp. 956

Author(s):

Steve McDonald ◽

Julian H. Elliott ◽

Sally Green ◽

Tari Turner

Keyword(s):

Systematic Reviews ◽

New Technologies ◽

Research Output ◽

Guideline Development ◽

Evidence Based ◽

Structured Interviews ◽

Australian Guideline ◽

Horizon Scanning ◽

New Guidelines ◽

Evidence Based Guidelines

Background: Many organisations in Australia undertake systematic reviews to inform development of evidence-based guidelines or would like to do so. However, the substantial resources required to produce systematic reviews limit the feasibility of evidence-based approaches to guideline development. We are working with Australian guideline developers to design, build and test systems that make creating evidence-based guidelines easier and more efficient. Methods: To understand the evidence needs of guideline developers and to inform the development of potential tools and services, we conducted 16 semi-structured interviews with Australian guideline developers. Developers were involved in different types of guidelines, represented both new and established guideline groups, and had access to widely different levels of resources. Results: All guideline developers recognised the importance of having access to timely evidence to support their processes, but were frequently overwhelmed by the scale of this task. Groups developing new guidelines often underestimated the time, expertise and work involved in completing searching and screening. Many were grappling with the challenge of updating and were keen to explore alternatives to the blanket updating of the full guideline. Horizon-scanning and evidence signalling were seen as providing more pragmatic approaches to updating, although some were wary of challenges posed by receiving evidence on a too-frequent basis. Respondents were aware that new technologies, such as machine learning, offered potentially large time and resource savings. Conclusions: As well as the constant challenge of managing financial constraints, Australian guideline developers seeking to develop clinical guidelines face several critical challenges. These include acquiring appropriate methodological expertise, investing in information technology, coping with the proliferation of research output, feasible publication and dissemination options, and keeping guidance up to date.

Download Full-text

Taking account of context in systematic reviews and guidelines considering a complexity perspective

BMJ Global Health ◽

10.1136/bmjgh-2018-000840 ◽

2019 ◽

Vol 4 (Suppl 1) ◽

pp. e000840 ◽

Cited By ~ 24

Author(s):

Andrew Booth ◽

Graham Moore ◽

Kate Flemming ◽

Ruth Garside ◽

Nigel Rollins ◽

...

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Guideline Development ◽

Evidence Synthesis ◽

Complex Interventions ◽

Target Populations ◽

Contextual Variation ◽

Rich Diversity ◽

Contextual Detail ◽

Systematic Review Process

Systematic review teams and guideline development groups face considerable challenges when considering context within the evidence production process. Many complex interventions are context-dependent and are frequently evaluated within considerable contextual variation and change. This paper considers the extent to which current tools used within systematic reviews and guideline development are suitable in meeting these challenges. The paper briefly reviews strengths and weaknesses of existing approaches to specifying context. Illustrative tools are mapped to corresponding stages of the systematic review process. Collectively, systematic review and guideline production reveals a rich diversity of frameworks and tools for handling context. However, current approaches address only specific elements of context, are derived from primary studies which lack information or have not been tested within systematic reviews. A hypothetical example is used to illustrate how context could be integrated throughout the guideline development process. Guideline developers and evidence synthesis organisations should select an appropriate level of contextual detail for their specific guideline that is parsimonious and yet sensitive to health systems contexts and the values, preferences and needs of their target populations.

Download Full-text

Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools

Systematic Reviews ◽

10.1186/s13643-019-1222-2 ◽

2019 ◽

Vol 8 (1) ◽

Cited By ~ 3

Author(s):

Allison Gates ◽

Samantha Guitard ◽

Jennifer Pillay ◽

Sarah A. Elliott ◽

Michele P. Dyson ◽

...

Keyword(s):

Machine Learning ◽

Systematic Reviews ◽

Relative Advantage ◽

Learning Tools ◽

User Friendliness ◽

User Experiences ◽

Increased Risk ◽

Independent Screening ◽

Automated Simulation ◽

Time Savings

Abstract Background We explored the performance of three machine learning tools designed to facilitate title and abstract screening in systematic reviews (SRs) when used to (a) eliminate irrelevant records (automated simulation) and (b) complement the work of a single reviewer (semi-automated simulation). We evaluated user experiences for each tool. Methods We subjected three SRs to two retrospective screening simulations. In each tool (Abstrackr, DistillerSR, RobotAnalyst), we screened a 200-record training set and downloaded the predicted relevance of the remaining records. We calculated the proportion missed and workload and time savings compared to dual independent screening. To test user experiences, eight research staff tried each tool and completed a survey. Results Using Abstrackr, DistillerSR, and RobotAnalyst, respectively, the median (range) proportion missed was 5 (0 to 28) percent, 97 (96 to 100) percent, and 70 (23 to 100) percent for the automated simulation and 1 (0 to 2) percent, 2 (0 to 7) percent, and 2 (0 to 4) percent for the semi-automated simulation. The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation and 40 (32 to 43) percent, 49 (48 to 49) percent, and 35 (34 to 38) percent for the semi-automated simulation. The median (range) time savings was 154 (91 to 183), 185 (95 to 201), and 157 (86 to 172) hours for the automated simulation and 61 (42 to 82), 92 (46 to 100), and 64 (37 to 71) hours for the semi-automated simulation. Abstrackr identified 33–90% of records missed by a single reviewer. RobotAnalyst performed less well and DistillerSR provided no relative advantage. User experiences depended on user friendliness, qualities of the user interface, features and functions, trustworthiness, ease and speed of obtaining predictions, and practicality of the export file(s). Conclusions The workload savings afforded in the automated simulation came with increased risk of missing relevant records. Supplementing a single reviewer’s decisions with relevance predictions (semi-automated simulation) sometimes reduced the proportion missed, but performance varied by tool and SR. Designing tools based on reviewers’ self-identified preferences may improve their compatibility with present workflows. Systematic review registration Not applicable.

Download Full-text

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Systematic Reviews ◽

10.1186/s13643-021-01635-3 ◽

2021 ◽

Vol 10 (1) ◽

Author(s):

Kevin E. K. Chai ◽

Robin L. J. Lines ◽

Daniel F. Gucciardi ◽

Leo Ng

Keyword(s):

Machine Learning ◽

Systematic Reviews ◽

Real World ◽

New Technologies ◽

Learning Tool ◽

Level Of Evidence ◽

Time Saving ◽

Scoping Reviews ◽

Abstract Screening ◽

Machine Learning Tool

Abstract Background Systematic reviews and meta-analyses provide the highest level of evidence to help inform policy and practice, yet their rigorous nature is associated with significant time and economic demands. The screening of titles and abstracts is the most time consuming part of the review process with analysts required review thousands of articles manually, taking on average 33 days. New technologies aimed at streamlining the screening process have provided initial promising findings, yet there are limitations with current approaches and barriers to the widespread use of these tools. In this paper, we introduce and report initial evidence on the utility of Research Screener, a semi-automated machine learning tool to facilitate abstract screening. Methods Three sets of analyses (simulation, interactive and sensitivity) were conducted to provide evidence of the utility of the tool through both simulated and real-world examples. Results Research Screener delivered a workload saving of between 60 and 96% across nine systematic reviews and two scoping reviews. Findings from the real-world interactive analysis demonstrated a time saving of 12.53 days compared to the manual screening, which equates to a financial saving of USD 2444. Conservatively, our results suggest that analysts who scan 50% of the total pool of articles identified via a systematic search are highly likely to have identified 100% of eligible papers. Conclusions In light of these findings, Research Screener is able to reduce the burden for researchers wishing to conduct a comprehensive systematic review without reducing the scientific rigour for which they strive to achieve.

Download Full-text

Evolving robotic surgery training and improving patient safety, with the integration of novel technologies

World Journal of Urology ◽

10.1007/s00345-020-03467-7 ◽

2020 ◽

Author(s):

I-Hsuan Alan Chen ◽

Ahmed Ghazi ◽

Ashwin Sridhar ◽

Danail Stoyanov ◽

Mark Slack ◽

...

Keyword(s):

Machine Learning ◽

Robotic Surgery ◽

Performance Feedback ◽

New Technologies ◽

Current Evidence ◽

Robotic Training ◽

Novel Technologies ◽

Surgery Training ◽

Consensus Statements ◽

Robotic Surgery Training

Abstract Introduction Robot-assisted surgery is becoming increasingly adopted by multiple surgical specialties. There is evidence of inherent risks of utilising new technologies that are unfamiliar early in the learning curve. The development of standardised and validated training programmes is crucial to deliver safe introduction. In this review, we aim to evaluate the current evidence and opportunities to integrate novel technologies into modern digitalised robotic training curricula. Methods A systematic literature review of the current evidence for novel technologies in surgical training was conducted online and relevant publications and information were identified. Evaluation was made on how these technologies could further enable digitalisation of training. Results Overall, the quality of available studies was found to be low with current available evidence consisting largely of expert opinion, consensus statements and small qualitative studies. The review identified that there are several novel technologies already being utilised in robotic surgery training. There is also a trend towards standardised validated robotic training curricula. Currently, the majority of the validated curricula do not incorporate novel technologies and training is delivered with more traditional methods that includes centralisation of training services with wet laboratories that have access to cadavers and dedicated training robots. Conclusions Improvements to training standards and understanding performance data have good potential to significantly lower complications in patients. Digitalisation automates data collection and brings data together for analysis. Machine learning has potential to develop automated performance feedback for trainees. Digitalised training aims to build on the current gold standards and to further improve the ‘continuum of training’ by integrating PBP training, 3D-printed models, telementoring, telemetry and machine learning.

Download Full-text