Digging for the truth: the case for active annotation in evaluating the credibility of online medical information (Preprint)

2020 ◽  
Author(s):  
Mikołaj Morzy ◽  
Bartłomiej Balcerzak ◽  
Adam Wierzbicki ◽  
Adam Wierzbicki
Keyword(s):  
Machine Learning ◽  
Medical Information ◽  
Representation Learning ◽  
Training Dataset ◽  
Highly Qualified ◽  
Human In The Loop ◽  
Annotation Process ◽  
Comprehensive Framework ◽  
Online Sources ◽  
The Web

BACKGROUND With the rapidly accelerating spread of dissemination of false medical information on the Web, the task of establishing the credibility of online sources of medical information becomes a pressing necessity. The sheer number of websites offering questionable medical information presented as reliable and actionable suggestions with possibly harmful effects poses an additional requirement for potential solutions, as they have to scale to the size of the problem. Machine learning is one such solution which, when properly deployed, can be an effective tool in fighting medical disinformation on the Web. OBJECTIVE We present a comprehensive framework for designing and curating of machine learning training datasets for online medical information credibility assessment. We show how the annotation process should be constructed and what pitfalls should be avoided. Our main objective is to provide researchers from medical and computer science communities with guidelines on how to construct datasets for machine learning models for various areas of medical information wars. METHODS The key component of our approach is the active annotation process. We begin by outlining the annotation protocol for the curation of high-quality training dataset, which then can be augmented and rapidly extended by employing the human-in-the-loop paradigm to machine learning training. To circumvent the cold start problem of insufficient gold standard annotations, we propose a pre-processing pipeline consisting of representation learning, clustering, and re-ranking of sentences for the acceleration of the training process and the optimization of human resources involved in the annotation. RESULTS We collect over 10 000 annotations of sentences related to selected subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, food allergy testing) for less than $7 000 employing 9 highly qualified annotators (certified medical professionals) and we release this dataset to the general public. We develop an active annotation framework for more efficient annotation of non-credible medical statements. The results of the qualitative analysis support our claims of the efficacy of the presented method. CONCLUSIONS A set of very diverse incentives is driving the widespread dissemination of medical disinformation on the Web. An effective strategy of countering this spread is to use machine learning for automatically establishing the credibility of online medical information. This, however, requires a thoughtful design of the training pipeline. In this paper we present a comprehensive framework of active annotation. In addition, we publish a large curated dataset of medical statements labelled as credible, non-credible, or neutral.

Digging for the truth: the case for active annotation in evaluating the credibility of online medical information (Preprint)

2020 ◽  
Author(s):  
Aleksandra Nabożny ◽  
Bartłomiej Balcerzak ◽  
Adam Wierzbicki ◽  
Mikołaj Morzy
Keyword(s):  
Machine Learning ◽  
Medical Information ◽  
Representation Learning ◽  
Training Dataset ◽  
Highly Qualified ◽  
Human In The Loop ◽  
Annotation Process ◽  
Comprehensive Framework ◽  
Online Sources ◽  
The Web

BACKGROUND With the rapidly accelerating spread of dissemination of false medical information on the Web, the task of establishing the credibility of online sources of medical information becomes a pressing necessity. The sheer number of websites offering questionable medical information presented as reliable and actionable suggestions with possibly harmful effects poses an additional requirement for potential solutions, as they have to scale to the size of the problem. Machine learning is one such solution which, when properly deployed, can be an effective tool in fighting medical disinformation on the Web. OBJECTIVE We present a comprehensive framework for designing and curating of machine learning training datasets for online medical information credibility assessment. We show how the annotation process should be constructed and what pitfalls should be avoided. Our main objective is to provide researchers from medical and computer science communities with guidelines on how to construct datasets for machine learning models for various areas of medical information wars. METHODS The key component of our approach is the active annotation process. We begin by outlining the annotation protocol for the curation of high-quality training dataset, which then can be augmented and rapidly extended by employing the human-in-the-loop paradigm to machine learning training. To circumvent the cold start problem of insufficient gold standard annotations, we propose a pre-processing pipeline consisting of representation learning, clustering, and re-ranking of sentences for the acceleration of the training process and the optimization of human resources involved in the annotation. RESULTS We collect over 10 000 annotations of sentences related to selected subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, food allergy testing) for less than $7 000 employing 9 highly qualified annotators (certified medical professionals) and we release this dataset to the general public. We develop an active annotation framework for more efficient annotation of non-credible medical statements. The results of the qualitative analysis support our claims of the efficacy of the presented method. CONCLUSIONS A set of very diverse incentives is driving the widespread dissemination of medical disinformation on the Web. An effective strategy of countering this spread is to use machine learning for automatically establishing the credibility of online medical information. This, however, requires a thoughtful design of the training pipeline. In this paper we present a comprehensive framework of active annotation. In addition, we publish a large curated dataset of medical statements labelled as credible, non-credible, or neutral.

scholarly journals eXplainable Cooperative Machine Learning with NOVA

2020 ◽  
Vol 34 (2) ◽  
pp. 143-164 ◽  
Author(s):  
Tobias Baur ◽  
Alexander Heimerl ◽  
Florian Lingenfelser ◽  
Johannes Wagner ◽  
Michel F. Valstar ◽  
...  
Keyword(s):  
Machine Learning ◽  
Main Idea ◽  
Data Annotation ◽  
Training Tool ◽  
Human In The Loop ◽  
Case Evaluation ◽  
Annotation Process ◽  
Multiple Annotators ◽  
Speed Up ◽  
Model Training

Abstract In the following article, we introduce a novel workflow, which we subsume under the term “explainable cooperative machine learning” and show its practical application in a data annotation and model training tool called NOVA. The main idea of our approach is to interactively incorporate the ‘human in the loop’ when training classification models from annotated data. In particular, NOVA offers a collaborative annotation backend where multiple annotators join their workforce. A main aspect is the possibility of applying semi-supervised active learning techniques already during the annotation process by giving the possibility to pre-label data automatically, resulting in a drastic acceleration of the annotation process. Furthermore, the user-interface implements recent eXplainable AI techniques to provide users with both, a confidence value of the automatically predicted annotations, as well as visual explanation. We show in an use-case evaluation that our workflow is able to speed up the annotation process, and further argue that by providing additional visual explanations annotators get to understand the decision making process as well as the trustworthiness of their trained machine learning models.

scholarly journals Novel Surveillance of Occupational Injury Powered by Machine Learning Using Chief Complaint at Emergency Triage

2020 ◽  
Author(s):  
Ting-Chia Weng ◽  
Yu-Ting Wei ◽  
Tsung-Yu Chan ◽  
Wen-Chau Chen ◽  
Ming-Hong Chen ◽  
...  
Keyword(s):  
Machine Learning ◽  
Language Processing ◽  
Medical Information ◽  
Occupational Injuries ◽  
Medical Center ◽  
Chief Complaint ◽  
Supervised Machine Learning ◽  
Training Dataset ◽  
Reimbursement Rate ◽  
Chief Complaints

Abstract Background: Underreporting of occupational injuries remains a global health issue, and warrants social awareness. To assist surveillance of occupational injuries, we developed an automatic screening model using chief complaint at emergency department to improve detection and reporting of occupational injuries.Methods: A total of 181,843 emergency visits aged 15 to 65 year-old were included at a medical center in Taiwan from April, 2015 to March, 2018. The retrospective cohort composed of 9,307 cases of occupational injuries and 172,536 controls. We applied the first 30 months as training dataset, and the following 6 months for prospective testing. Natural Language Processing (NLP) was applied to analyze patients’ chief complaints. The sentences were processed by JiebaR, a Chinese text segmentation technique, reviewed by two occupational physicians, and transformed by a word-embedding model using Glove. Logistic regression was conducted to predict suspected cases of occupational injuries.Results: The prediction model using the chief complaint alone can achieve an overall AUC of 0.936 [95% confidence interval (CI): 0.931 to 0.941], with high sensitivity (93.1%, 95% CI: 91.8 to 94.4) and adequate specificity (84.8%, 95% CI: 84.4 to 85.2). Patients in the most urgent or severe conditions showed the highest accuracy around 90%. We also observed increasing trends of referral to specialists of occupational medicine, and reimbursement rate.Conclusions: Interdepartmental coordination and integration of machine learning may augment the detection of occupational injuries at the emergency triage and improve the reporting, compensation, and prevention of occupational injuries.Contributions to the literaturel This novel surveillance powered by supervised machine learning can help to identify occupational injuries at the emergency triage, and it achieves 93.1% sensitivity and 84.8% specificity.l Along with the system implementation, there are trends of increasing reimbursement rate in the National Labor Insurance and increasing case referrals to specialists of occupational medicine.l Combining the use of Artificial Intelligence (AI) with conventional medical information, such as chief complaints, may provide an effective surveillance system for occupational injuries at hospitals. l Effective screening of labors who suffer from occupational injuries will promote Total Worker Health.

Comparison of Machine Learning Algorithms in Predicting the COVID-19 Outbreak

2022 ◽  
pp. 320-336
Author(s):  
Asiye Bilgili
Keyword(s):  
Machine Learning ◽  
Performance Evaluation ◽  
Health Informatics ◽  
Medical Information ◽  
Learning Algorithms ◽  
Model Performance ◽  
Machine Learning Algorithms ◽  
Training Dataset ◽  
Implementation Phase ◽  
Model Performance Evaluation

Health informatics is an interdisciplinary field in the computer and health sciences. Health informatics, which enables the effective use of medical information, has the potential to reduce both the cost and the burden of healthcare workers during the pandemic process. Using the machine learning algorithms support vector machines, naive bayes, k-nearest neighbor, and C4.5 algorithms, a model performance evaluation was performed to identify the algorithm that will show the highest performance for the prediction of the disease. Three separate training and test datasets were created 70% - 30%, 75% - 25%, and 80% - 20%, respectively. The implementation phase of the study was carried out by following the CRISP-DM steps, and the analyses were made using the R language. By examining the model performance evaluation criteria, the findings show that the C4.5 algorithm showed the best performance with 70% training dataset.

Supervised Classifier Approach for Intrusion Detection on KDD with Optimal MapReduce Framework Model in Cloud Computing

2019 ◽  
Vol 12 ◽  
Author(s):  
M. Ilayaraja ◽  
S. Hemalatha ◽  
P. Manickam ◽  
K. Sathesh Kumar ◽  
K. Shankar
Keyword(s):  
Machine Learning ◽  
Cloud Computing ◽  
Intrusion Detection ◽  
Decision Tree ◽  
Learning Strategies ◽  
Nearest Neighbor ◽  
Detection System ◽  
K Nearest Neighbor ◽  
Mapreduce Model ◽  
The Web

Cloud computing is characterized as the arrangement of assets or administrations accessible through the web to the clients on their request by cloud providers. It communicates everything as administrations over the web in view of the client request, for example operating system, organize equipment, storage, assets, and software. Nowadays, Intrusion Detection System (IDS) plays a powerful system, which deals with the influence of experts to get actions when the system is hacked under some intrusions. Most intrusion detection frameworks are created in light of machine learning strategies. Since the datasets, this utilized as a part of intrusion detection is Knowledge Discovery in Database (KDD). In this paper detect or classify the intruded data utilizing Machine Learning (ML) with the MapReduce model. The primary face considers Hadoop MapReduce model to reduce the extent of database ideal weight decided for reducer model and second stage utilizing Decision Tree (DT) classifier to detect the data. This DT classifier comprises utilizing an appropriate classifier to decide the class labels for the non-homogeneous leaf nodes. The decision tree fragment gives a coarse section profile while the leaf level classifier can give data about the qualities that influence the label inside a portion. From the proposed result accuracy for detection is 96.21% contrasted with existing classifiers, for example, Neural Network (NN), Naive Bayes (NB) and K Nearest Neighbor (KNN).

AN EFFICIENT MACHINE LEARNING MODEL FOR PREDICTION OF ACUTE MYOCARDIAL INFARCTION

2020 ◽  
Vol 13 ◽  
Author(s):  
Dhilsath Fathima.M ◽  
S. Justin Samuel ◽  
R. Hari Haran
Keyword(s):  
Machine Learning ◽  
Myocardial Infarction ◽  
Acute Myocardial Infarction ◽  
Logistic Regression ◽  
Decision Tree ◽  
Learning Model ◽  
Training Dataset ◽  
Data Set ◽  
Machine Learning Model ◽  
Proposed Model

Aim: This proposed work is used to develop an improved and robust machine learning model for predicting Myocardial Infarction (MI) could have substantial clinical impact. Objectives: This paper explains how to build machine learning based computer-aided analysis system for an early and accurate prediction of Myocardial Infarction (MI) which utilizes framingham heart study dataset for validation and evaluation. This proposed computer-aided analysis model will support medical professionals to predict myocardial infarction proficiently. Methods: The proposed model utilize the mean imputation to remove the missing values from the data set, then applied principal component analysis to extract the optimal features from the data set to enhance the performance of the classifiers. After PCA, the reduced features are partitioned into training dataset and testing dataset where 70% of the training dataset are given as an input to the four well-liked classifiers as support vector machine, k-nearest neighbor, logistic regression and decision tree to train the classifiers and 30% of test dataset is used to evaluate an output of machine learning model using performance metrics as confusion matrix, classifier accuracy, precision, sensitivity, F1-score, AUC-ROC curve. Results: Output of the classifiers are evaluated using performance measures and we observed that logistic regression provides high accuracy than K-NN, SVM, decision tree classifiers and PCA performs sound as a good feature extraction method to enhance the performance of proposed model. From these analyses, we conclude that logistic regression having good mean accuracy level and standard deviation accuracy compared with the other three algorithms. AUC-ROC curve of the proposed classifiers is analyzed from the output figure.4, figure.5 that logistic regression exhibits good AUC-ROC score, i.e. around 70% compared to k-NN and decision tree algorithm. Conclusion: From the result analysis, we infer that this proposed machine learning model will act as an optimal decision making system to predict the acute myocardial infarction at an early stage than an existing machine learning based prediction models and it is capable to predict the presence of an acute myocardial Infarction with human using the heart disease risk factors, in order to decide when to start lifestyle modification and medical treatment to prevent the heart disease.

Derivation of Respiratory Metrics in Health and Asthma; Machine Learning Methodology (Preprint)

2020 ◽  
Author(s):  
Joseph Prinable ◽  
Peter Jones ◽  
David Boland ◽  
Alistair McEwan ◽  
Cindy Thamrin
Keyword(s):  
Machine Learning ◽  
Short Term Memory ◽  
Reference Signal ◽  
Machine Learning Algorithms ◽  
Training Dataset ◽  
Local Health ◽  
Strongly Correlated ◽  
Human Research Ethics ◽  
Expiration Time ◽  
Wide Variability

BACKGROUND The ability to continuously monitor breathing metrics may have indications for general health as well as respiratory conditions such as asthma. However, few studies have focused on breathing due to a lack of available wearable technologies. OBJECTIVE Examine the performance of two machine learning algorithms in extracting breathing metrics from a finger-based pulse oximeter, which is amenable to long-term monitoring. METHODS Pulse oximetry data was collected from 11 healthy and 11 asthma subjects who breathed at a range of controlled respiratory rates. UNET and Long Short-Term memory (LSTM) algorithms were applied to the data, and results compared against breathing metrics derived from respiratory inductance plethysmography measured simultaneously as a reference. RESULTS The UNET vs LSTM model provided breathing metrics which were strongly correlated with those from the reference signal (all p<0.001, except for inspiratory:expiratory ratio). The following relative mean bias(95% confidence interval) were observed: inspiration time 1.89(-52.95, 56.74)% vs 1.30(-52.15, 54.74)%, expiration time -3.70(-55.21, 47.80)% vs -4.97(-56.84, 46.89)%, inspiratory:expiratory ratio -4.65(-87.18, 77.88)% vs -5.30(-87.07, 76.47)%, inter-breath intervals -2.39(-32.76, 27.97)% vs -3.16(-33.69, 27.36)%, and respiratory rate 2.99(-27.04 to 33.02)% vs 3.69(-27.17 to 34.56)%. CONCLUSIONS Both machine learning models show strongly correlation and good comparability with reference, with low bias though wide variability for deriving breathing metrics in asthma and health cohorts. Future efforts should focus on improvement of performance of these models, e.g. by increasing the size of the training dataset at the lower breathing rates. CLINICALTRIAL Sydney Local Health District Human Research Ethics Committee (#LNR\16\HAWKE99 ethics approval).

scholarly journals A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

2021 ◽  
Vol 18 (4) ◽  
pp. 2121
Author(s):  
Mansoureh Maadi ◽  
Hadi Akbarzadeh Khorshidi ◽  
Uwe Aickelin
Keyword(s):  
Machine Learning ◽  
Future Research ◽  
Computational Power ◽  
Medical Field ◽  
Interactive Machine Learning ◽  
Human In The Loop ◽  
Human Interactions ◽  
Scoping Literature Review ◽  
Domain Expertise ◽  
Expertise Level

Objective: To provide a human–Artificial Intelligence (AI) interaction review for Machine Learning (ML) applications to inform how to best combine both human domain expertise and computational power of ML methods. The review focuses on the medical field, as the medical ML application literature highlights a special necessity of medical experts collaborating with ML approaches. Methods: A scoping literature review is performed on Scopus and Google Scholar using the terms “human in the loop”, “human in the loop machine learning”, and “interactive machine learning”. Peer-reviewed papers published from 2015 to 2020 are included in our review. Results: We design four questions to investigate and describe human–AI interaction in ML applications. These questions are “Why should humans be in the loop?”, “Where does human–AI interaction occur in the ML processes?”, “Who are the humans in the loop?”, and “How do humans interact with ML in Human-In-the-Loop ML (HILML)?”. To answer the first question, we describe three main reasons regarding the importance of human involvement in ML applications. To address the second question, human–AI interaction is investigated in three main algorithmic stages: 1. data producing and pre-processing; 2. ML modelling; and 3. ML evaluation and refinement. The importance of the expertise level of the humans in human–AI interaction is described to answer the third question. The number of human interactions in HILML is grouped into three categories to address the fourth question. We conclude the paper by offering a discussion on open opportunities for future research in HILML.

scholarly journals Exploratory Analysis of Driving Force of Wildfires in Australia: An Application of Machine Learning within Google Earth Engine

2020 ◽  
Vol 13 (1) ◽  
pp. 10
Author(s):  
Andrea Sulova ◽  
Jamal Jokar Arsanjani
Keyword(s):  
Climate Change ◽  
Machine Learning ◽  
Random Forest ◽  
Google Earth ◽  
Summer Season ◽  
Driving Factors ◽  
Machine Learning Algorithms ◽  
Classification And Regression Tree ◽  
Training Dataset ◽  
Google Earth Engine

Recent studies have suggested that due to climate change, the number of wildfires across the globe have been increasing and continue to grow even more. The recent massive wildfires, which hit Australia during the 2019–2020 summer season, raised questions to what extent the risk of wildfires can be linked to various climate, environmental, topographical, and social factors and how to predict fire occurrences to take preventive measures. Hence, the main objective of this study was to develop an automatized and cloud-based workflow for generating a training dataset of fire events at a continental level using freely available remote sensing data with a reasonable computational expense for injecting into machine learning models. As a result, a data-driven model was set up in Google Earth Engine platform, which is publicly accessible and open for further adjustments. The training dataset was applied to different machine learning algorithms, i.e., Random Forest, Naïve Bayes, and Classification and Regression Tree. The findings show that Random Forest outperformed other algorithms and hence it was used further to explore the driving factors using variable importance analysis. The study indicates the probability of fire occurrences across Australia as well as identifies the potential driving factors of Australian wildfires for the 2019–2020 summer season. The methodical approach and achieved results and drawn conclusions can be of great importance to policymakers, environmentalists, and climate change researchers, among others.

scholarly journals Deep Learning Classification of Canine Behavior Using a Single Collar-Mounted Accelerometer: Real-World Validation

Animals ◽  
2021 ◽  
Vol 11 (6) ◽  
pp. 1549
Author(s):  
Robert D. Chambers ◽  
Nathanael C. Yoder ◽  
Aletha B. Carson ◽  
Christian Junge ◽  
David E. Allen ◽  
...  
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Real World ◽  
Learning Algorithm ◽  
Drinking Behavior ◽  
True Positive Rate ◽  
Training Dataset ◽  
Activity Levels ◽  
Accelerometer Data ◽  
Activity Monitors

Collar-mounted canine activity monitors can use accelerometer data to estimate dog activity levels, step counts, and distance traveled. With recent advances in machine learning and embedded computing, much more nuanced and accurate behavior classification has become possible, giving these affordable consumer devices the potential to improve the efficiency and effectiveness of pet healthcare. Here, we describe a novel deep learning algorithm that classifies dog behavior at sub-second resolution using commercial pet activity monitors. We built machine learning training databases from more than 5000 videos of more than 2500 dogs and ran the algorithms in production on more than 11 million days of device data. We then surveyed project participants representing 10,550 dogs, which provided 163,110 event responses to validate real-world detection of eating and drinking behavior. The resultant algorithm displayed a sensitivity and specificity for detecting drinking behavior (0.949 and 0.999, respectively) and eating behavior (0.988, 0.983). We also demonstrated detection of licking (0.772, 0.990), petting (0.305, 0.991), rubbing (0.729, 0.996), scratching (0.870, 0.997), and sniffing (0.610, 0.968). We show that the devices’ position on the collar had no measurable impact on performance. In production, users reported a true positive rate of 95.3% for eating (among 1514 users), and of 94.9% for drinking (among 1491 users). The study demonstrates the accurate detection of important health-related canine behaviors using a collar-mounted accelerometer. We trained and validated our algorithms on a large and realistic training dataset, and we assessed and confirmed accuracy in production via user validation.

Sign in / Sign up

Export Citation Format

Share Document