Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses

Abstract Background Systematic reviews are the cornerstone of evidence-based medicine. However, systematic reviews are time consuming and there is growing demand to produce evidence more quickly, while maintaining robust methods. In recent years, artificial intelligence and active-machine learning (AML) have been implemented into several SR software applications. As some of the barriers to adoption of new technologies are the challenges in set-up and how best to use these technologies, we have provided different situations and considerations for knowledge synthesis teams to consider when using artificial intelligence and AML for title and abstract screening. Methods We retrospectively evaluated the implementation and performance of AML across a set of ten historically completed systematic reviews. Based upon the findings from this work and in consideration of the barriers we have encountered and navigated during the past 24 months in using these tools prospectively in our research, we discussed and developed a series of practical recommendations for research teams to consider in seeking to implement AML tools for citation screening into their workflow. Results We developed a seven-step framework and provide guidance for when and how to integrate artificial intelligence and AML into the title and abstract screening process. Steps include: (1) Consulting with Knowledge user/Expert Panel; (2) Developing the search strategy; (3) Preparing your review team; (4) Preparing your database; (5) Building the initial training set; (6) Ongoing screening; and (7) Truncating screening. During Step 6 and/or 7, you may also choose to optimize your team, by shifting some members to other review stages (e.g., full-text screening, data extraction). Conclusion Artificial intelligence and, more specifically, AML are well-developed tools for title and abstract screening and can be integrated into the screening process in several ways. Regardless of the method chosen, transparent reporting of these methods is critical for future studies evaluating artificial intelligence and AML.

Download Full-text

Clinical handover and handoff in healthcare: a systematic review of systematic reviews

International Journal for Quality in Health Care ◽

10.1093/intqhc/mzaa170 ◽

2020 ◽

Author(s):

Melissa Desmedt ◽

Dorien Ulenaers ◽

Joep Grosemans ◽

Johan Hellings ◽

Jochen Bergs

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Data Extraction ◽

Implementation Process ◽

Healthcare Setting ◽

Well Being ◽

Role Playing ◽

Included Study ◽

Clinical Handover ◽

Review Team

Abstract Purpose The purpose of this systematic review is to appraise and summarize existing literature on clinical handover. Data sources We searched EMBASE, MEDLINE, Database of Abstracts of Reviews of Effects and Cochrane Database of Systematic Reviews. Study selection Included articles were reviewed independently by the review team. Data extraction The review team extracted data under the following headers: author(s), year of publication, journal, scope, search strategy, number of studies included, type of studies included, study quality assessment, used definition of handover, healthcare setting, outcomes measured, findings and finally some comments or remarks. Results of data synthesis First, research indicates that poor handover is associated with multiple potential hazards such as lack of availability of required equipment for patients, information omissions, diagnosis errors, treatment errors, disposition errors and treatment delays. Second, our systematic review indicates that no single tool arises as best for any particular specialty or use to evaluate the handover process. Third, there is little evidence delineating what constitutes best handoff practices. Most efforts facilitated the coordination of care and communication between healthcare professionals using electronic tools or a standardized form. Fourth, our review indicates that the principal teaching methods are role-playing and simulation, which may result in better knowledge transfer to the work environment, better health and patients’ well-being. Conclusions This review emphasizes the importance of staff education (including simulation-based and team training), non-technical skills and the implementation process of clinical handover in healthcare settings.

Download Full-text

A Research Protocol for a Systematic Review of Automatic Literature Screening in Medical Evidence Synthesis

10.21203/rs.3.rs-62316/v1 ◽

2020 ◽

Author(s):

Yuelun Zhang ◽

Siyu Liang ◽

Yunying Feng ◽

Qing Wang ◽

Feng Sun ◽

...

Keyword(s):

Systematic Review ◽

Systematic Reviews ◽

Data Extraction ◽

Evidence Synthesis ◽

Evidence Based ◽

Medical Evidence ◽

Registration Number ◽

Evidence Collection ◽

Automatic Methods ◽

And Performance

Abstract Background: Systematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies.Methods: An electronic search will be conducted using PubMed, Embase and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed.Discussion: Automating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis, and help to inspire further studies in this field. Registration: PROSPERO registration number CRD42020170815 (28 April 2020).

Download Full-text

Implementation of Artificial Intelligence in Fashion: Are Consumers Ready?

Clothing and Textiles Research Journal ◽

10.1177/0887302x19873437 ◽

2019 ◽

Vol 38 (1) ◽

pp. 3-18 ◽

Cited By ~ 2

Author(s):

Yuli Liang ◽

Seung-Hee Lee ◽

Jane E. Workman

Keyword(s):

Artificial Intelligence ◽

Technology Acceptance ◽

Purchase Intention ◽

New Technologies ◽

The United States ◽

Ease Of Use ◽

Perceived Usefulness ◽

Perceived Ease Of Use ◽

Positive Attitudes ◽

And Performance

Given the growing interest in combinations of fashion and digital innovations, it is critical for both researchers and retailers to understand how consumers respond to new technologies, especially artificial intelligence (AI). The purpose of the study was to examine consumers’ attitudes and purchase intention toward an AI device. By adapting the technology acceptance model, a conceptual model was constructed and tested related to consumers’ attitudes and purchase intention toward an AI device—Echo Look. A total of 313 subjects (61% female) between 18 and 65 years old in the top 10 metropolitan areas in the United States participated in the study. The results indicated that perceived usefulness, perceived ease of use, and performance risk were significant in consumers’ attitude toward AI. Positive attitudes toward technology positively influenced the purchase intention. Based on these results, theoretical and practical implications are discussed.

Download Full-text

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Systematic Reviews ◽

10.1186/s13643-021-01635-3 ◽

2021 ◽

Vol 10 (1) ◽

Author(s):

Kevin E. K. Chai ◽

Robin L. J. Lines ◽

Daniel F. Gucciardi ◽

Leo Ng

Keyword(s):

Machine Learning ◽

Systematic Reviews ◽

Real World ◽

New Technologies ◽

Learning Tool ◽

Level Of Evidence ◽

Time Saving ◽

Scoping Reviews ◽

Abstract Screening ◽

Machine Learning Tool

Abstract Background Systematic reviews and meta-analyses provide the highest level of evidence to help inform policy and practice, yet their rigorous nature is associated with significant time and economic demands. The screening of titles and abstracts is the most time consuming part of the review process with analysts required review thousands of articles manually, taking on average 33 days. New technologies aimed at streamlining the screening process have provided initial promising findings, yet there are limitations with current approaches and barriers to the widespread use of these tools. In this paper, we introduce and report initial evidence on the utility of Research Screener, a semi-automated machine learning tool to facilitate abstract screening. Methods Three sets of analyses (simulation, interactive and sensitivity) were conducted to provide evidence of the utility of the tool through both simulated and real-world examples. Results Research Screener delivered a workload saving of between 60 and 96% across nine systematic reviews and two scoping reviews. Findings from the real-world interactive analysis demonstrated a time saving of 12.53 days compared to the manual screening, which equates to a financial saving of USD 2444. Conservatively, our results suggest that analysts who scan 50% of the total pool of articles identified via a systematic search are highly likely to have identified 100% of eligible papers. Conclusions In light of these findings, Research Screener is able to reduce the burden for researchers wishing to conduct a comprehensive systematic review without reducing the scientific rigour for which they strive to achieve.

Download Full-text

An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes

BMC Medical Research Methodology ◽

10.1186/s12874-020-01129-1 ◽

2020 ◽

Vol 20 (1) ◽

Author(s):

C. Hamel ◽

S. E. Kelly ◽

K. Thavorn ◽

D. B. Rice ◽

G. A. Wells ◽

...

Keyword(s):

Artificial Intelligence ◽

Systematic Reviews ◽

Included Study ◽

Simulation Tool ◽

Rapid Reviews ◽

Screening Approach ◽

Response Sets ◽

The Hours ◽

Time Savings ◽

Abstract Screening

Abstract Background Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings. Methods Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented. Results Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%). Conclusions The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

Download Full-text

Innovative Devops for Artificial Intelligence

The Scientific Bulletin of Electrical Engineering Faculty ◽

10.1515/sbeef-2019-0011 ◽

2019 ◽

Vol 19 (1) ◽

pp. 58-63 ◽

Cited By ~ 1

Author(s):

R. Ciucu ◽

F.C. Adochiei ◽

Ioana-Raluca Adochiei ◽

F. Argatu ◽

G.C. Seriţan ◽

...

Keyword(s):

Artificial Intelligence ◽

Performance Optimization ◽

High Performance ◽

Computational Models ◽

Data Sets ◽

Production Environments ◽

Model Training ◽

And Performance ◽

Set Up ◽

Computational Resources

AbstractDeveloping Artificial Intelligence is a labor intensive task. It implies both storage and computational resources. In this paper, we present a state-of-the-art service based infrastructure for deploying, managing and serving computational models alongside their respective data-sets and virtual environments. Our architecture uses key-based values to store specific graphs and datasets into memory for fast deployment and model training, furthermore leveraging the need for manual data reduction in the drafting and retraining stages. To develop the platform, we used clustering and orchestration to set up services and containers that allow deployment within seconds. In this article, we cover high performance computing concepts such as swarming, GPU resource management for model implementation in production environments with emphasis on standardized development to reduce integration tasks and performance optimization.

Download Full-text

Factors underpinning discrepant findings from reviews on the same topic: A systematic analysis of Cochrane and non-Cochrane reviews

10.1101/574046 ◽

2019 ◽

Author(s):

Claudia Hacke ◽

David Nunan

Keyword(s):

Systematic Reviews ◽

Methodological Quality ◽

Data Extraction ◽

Meta Analysis ◽

List Type ◽

Eligibility Criteria ◽

Cochrane Reviews ◽

And Performance ◽

Meta Analyses

AbstractObjectiveTo explore factors underpinning discrepancies in reported pooled effect estimates from Cochrane and non-Cochrane systematic reviews answering the same question.Study Design and SettingWe observed discrepant pooled effects in 23 out of 24 pairs of meta-analyses from Cochrane and non-Cochrane systematic reviews answering the same question. Here we present the results of a systematic assessment of methodological quality and factors that explain the observed quantitative discrepancies. Methodological quality of each review was assessed using AMSTAR (Assessing the Methodological Quality of Systematic Reviews). Matched pairs were contrasted at the macro- (review methodology), meso- (application of methodology) and micro- (data extraction) level and reasons for differences were derived.ResultsAll Cochrane reviews had high methodological quality (AMSTAR 8-11), whereas the majority (87.5%) of non-Cochrane reviews were classified as moderate (AMSTAR 4-7). Only one pair included exactly the same studies for their respective meta-analyses but there was still a discrepancy in the pooled estimate due to differences in data extraction. One pair did not include any study of its match and for one pair the same effect estimates were reported despite inclusion of different studies. The remaining pairs included at least one study in their match. Due to insufficient reporting (predominantly affecting non-Cochrane reviews) we were only able to completely ascertain the reasons for discrepancies in all included studies for 9/24 (37.5%) pairs. Across all pairs, differences in pre-defined methods (macro-level) including search strategy, eligibility criteria and performance of dual screening could possibly explain mismatches in included studies. Study selection procedures (meso-level) including disagreements in the interpretation of pre-defined eligibility criteria (14 matches) were identified as reasons underpinning discrepant review findings. Comparison of data extraction from primary studies (micro-level) was not possible in 13/24 pairs as a result of the non-Cochrane review providing insufficient details of the studies included in their meta-analyses. Two out of 24 pairs completely agreed on the numerical data presented for the same studies in their respective meta-analysis. Both review types provided sufficient information to check the accuracy of data extraction for 8 pairs (45 studies) where there were discrepancies. An assessment of 50% (22 studies) of these showed that reasons for differences in extracted data could be identified in 15 studies. We found examples for both types of review where data presented were discrepant from that given in the source study without a plausible explanation.ConclusionMethodological and author judgements and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported pooled effect estimates between topic-matched reviews. Though caution must be taken when extrapolating, our findings raise the question as to what extent the entire meta-analysis evidence-base accurately reflects the available primary research both in terms of volume and data. Reinforcing awareness of the application of guidelines for systematic reviews and meta-analyses may help mitigate some of the key issues identified in our analysis.What is new?Key findings Non-Cochrane reviews were of a lower overall methodological quality compared with Cochrane reviews. Discrepant results of meta-analyses on the same topic can be attributed to differences in included studies based on review author decision, judgements and performance at different stages of the review process.What this adds to what was known?This study provides the most robust analysis to date of the potential methodological factors underpinning discrepant review findings between matched meta-analyses answering the same question. Assessing differences between reviews at the macro-, meso-, and micro-levels is a useful method to identify reasons for discrepant meta-analyses at key stages of the review process.What is the implication and what should change now?There is a need for a standardised approach to performing matched-pair analysis of meta-analyses and systematic reviews answering the same question. Our paper provides a base for this that can be refined by replication and expert consensus.

Download Full-text

Application of Information Technology Models, Approaches and Tools in Covid-19 Management: Rapid Review

Depiction of Health ◽

10.34172/doh.2021.09 ◽

2021 ◽

Vol 12 (1) ◽

pp. 77-95

Author(s):

Zeinab Mohammadzadeh ◽

Elham Maserat ◽

Reza Kariminezhad

Keyword(s):

Artificial Intelligence ◽

Information Technology ◽

Expert Systems ◽

New Technologies ◽

Data Extraction ◽

Economic Effects ◽

Rapid Screening ◽

Rapid Review ◽

Study Selection ◽

Complete Study

Background & Aims: Simultaneously with Covid-19 epidemic, much research has been done about to facilitate and expedite the diagnosis of disease, to establish vaccines and possible treatments, and to understand the socio-economic effects of this disease. New technologies such as telemedicine, ehealth, virtual reality, expert systems and artificial intelligence are very important in the progress and success of medical sciences. The experience of similar diseases such as MERS and SARS has confirmed the usefulness of information technology in management of infectious diseases and epidemics. The purpose of this study is to review the applications of information technology in Covid-19 management. Material & Methods: The present study is a five-step scoping review based on the proposed Arksey and O'Malley framework. a) Identifying the research objectives and search strategy b) Identifying relevant articles c) study selection d) data extraction and e) Summarizing, discussing, analyzing and reporting of results. Results: The 24 articles were selected for data extraction from 6,297 articles. The present study is the most complete study of technologies used in the field of Covid-19 management. Most studies (58%) have shown the ability of artificial intelligence to detect patients with Covid-19 and identify lesions using CT images. Other studies have confirmed the effectiveness of telemedicine systems, electronic health records, expert systems and monitoring systems to reduce contact, rapid screening and proper disease management & control. Also, 46% of related studies have been conducted in China. Conclusion: Due to high spreading capacity of the Covid-19, the use of telemedicine technologies has been more tangible in order to maintain social distance. Also, because of similarity of Covid-19 symptoms with other respiratory diseases, artificial intelligence applied to rapid screening and and diagnosis. The results of our study confirmed more application of these two systems. The results have proven the effectiveness of technology in quick patients identification, self-quarantine, diagnosis and monitoring of disease.

Download Full-text

Pathways ahead for New Zealand hill country farming

Journal of New Zealand Grasslands ◽

10.33584/jnzg.2016.78.522 ◽

2016 ◽

Vol 78 ◽

pp. 73-82 ◽

Cited By ~ 1

Author(s):

F.G. Scrimgeour

Keyword(s):

New Zealand ◽

Rural Communities ◽

New Technologies ◽

Service Providers ◽

Farming Systems ◽

Hill Country ◽

The Status ◽

And Performance ◽

Hill Farming ◽

The Hill

This paper provides a stocktake of the status of hill country farming in New Zealand and addresses the challenges which will determine its future state and performance. It arises out of the Hill Country Symposium, held in Rotorua, New Zealand, 12-13 April 2016. This paper surveys people, policy, business and change, farming systems for hill country, soil nutrients and the environment, plants for hill country, animals, animal feeding and productivity, and strategies for achieving sustainable outcomes in the hill country. This paper concludes by identifying approaches to: support current and future hill country farmers and service providers, to effectively and efficiently deal with change; link hill farming businesses to effective value chains and new markets to achieve sufficient and stable profitability; reward farmers for the careful management of natural resources on their farm; ensure that new technologies which improve the efficient use of input resources are developed; and strategies to achieve vibrant rural communities which strengthen hill country farming businesses and their service providers. Keywords: farming systems, hill country, people, policy, productivity, profitability, sustainability

Download Full-text

Artificial Intelligence as a Technological Innovation to Accelerate Economic Development

Economics taxes & law ◽

10.26794/1999-849x-2019-12-3-125-133 ◽

2019 ◽

Vol 12 (3) ◽

pp. 125-133

Author(s):

S. V. Shchurina ◽

A. S. Danilov

Keyword(s):

Artificial Intelligence ◽

Economic Development ◽

Technological Innovation ◽

Manufacturing Industry ◽

New Technologies ◽

Retail Trade ◽

High Tech ◽

Industrial Complex ◽

Technological Gap ◽

The Government

The subject of the research is the introduction of artificial intelligence as a technological innovation into the Russian economic development. The relevance of the problem is due to the fact that the Russian market of artificial intelligence is still in the infancy and the necessity to bridge the current technological gap between Russia and the leading economies of the world is coming to the forefront. The financial sector, the manufacturing industry and the retail trade are the drivers of the artificial intelligence development. However, company managers in Russia are not prepared for the practical application of expensive artificial intelligence technologies. Under these circumstances, the challenge is to develop measures to support high-tech projects of small and medium-sized businesses, given that the technological innovation considered can accelerate the development of the Russian economy in the energy sector fully or partially controlled by the government as well as in the military-industrial complex and the judicial system.The purposes of the research were to examine the current state of technological innovations in the field of artificial intelligence in the leading countries and Russia and develop proposals for improving the AI application in the Russian practices.The paper concludes that the artificial intelligence is a breakthrough technology with a great application potential. Active promotion of the artificial intelligence in companies significantly increases their efficiency, competitiveness, develops industry markets, stimulates introduction of new technologies, improves product quality and scales up manufacturing. In general, the artificial intelligence gives a new impetus to the development of Russia and facilitates its entry into the five largest world’s economies.

Download Full-text