Machine learning models predicting returns: why most popular performance metrics are misleading and proposal for an efficient metric

2021 ◽  
Author(s):  
Jean Dessain
Keyword(s):  
Machine Learning ◽  
Performance Metrics ◽  
Learning Models ◽  
Popular Performance ◽  
Machine Learning Models

Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning

2021 ◽  
Author(s):  
Chenxi Huang ◽  
Shu-Xia Li ◽  
César Caraballo ◽  
Frederick A. Masoudi ◽  
John S. Rumsfeld ◽  
...  
Keyword(s):  
Machine Learning ◽  
Risk Prediction ◽  
Health Care Professionals ◽  
Clinical Decision Making ◽  
Performance Metrics ◽  
Prediction Models ◽  
Learning Models ◽  
Risk Prediction Models ◽  
Clinical Risk ◽  
Machine Learning Models

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics. Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics. Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.

scholarly journals Assessment of Machine Learning Algorithms for Network Intrusion Detection

2020 ◽  
Vol 9 (4) ◽  
pp. 1667-1671
Keyword(s):  
Machine Learning ◽  
Intrusion Detection ◽  
Performance Metrics ◽  
Detection System ◽  
Machine Learning Algorithms ◽  
Network Intrusion Detection ◽  
Learning Models ◽  
Network Intrusion ◽  
Tree Classifier ◽  
Machine Learning Models

A Network Intrusion Detection System (NIDS) is a framework to identify network interruptions as well as abuse by checking network traffic movement and classifying it as either typical or strange. Numerous Intrusion Detection Systems have been implemented using simulated datasets like KDD’99 intrusion dataset but none of them uses a real time dataset. The proposed work performs and assesses tests to overview distinctive machine learning models reliant on KDD’99 intrusion dataset and an ongoing created dataset. The machine learning models achieved to compute required performance metrics so as to assess the chosen classifiers. The emphasis was on the accuracy metric so as to improve the recognition pace of the interruption identification framework. The actualized calculations showed that the decision tree classifier accomplished the most noteworthy estimation of accuracy while the logistic regression classifier has accomplished the least estimation of exactness for both of the datasets utilized.

scholarly journals Detection of Online Fake News Using Blending Ensemble Learning

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Arvin Hansrajh ◽  
Timothy T. Adeliyi ◽  
Jeanette Wing
Keyword(s):  
Machine Learning ◽  
Performance Metrics ◽  
Machine Learning Algorithms ◽  
Stochastic Gradient Descent ◽  
Support Vector ◽  
Fake News ◽  
Learning Models ◽  
Linear Discriminant ◽  
Proposed Model ◽  
Machine Learning Models

The exponential growth in fake news and its inherent threat to democracy, public trust, and justice has escalated the necessity for fake news detection and mitigation. Detecting fake news is a complex challenge as it is intentionally written to mislead and hoodwink. Humans are not good at identifying fake news. The detection of fake news by humans is reported to be at a rate of 54% and an additional 4% is reported in the literature as being speculative. The significance of fighting fake news is exemplified during the present pandemic. Consequently, social networks are ramping up the usage of detection tools and educating the public in recognising fake news. In the literature, it was observed that several machine learning algorithms have been applied to the detection of fake news with limited and mixed success. However, several advanced machine learning models are not being applied, although recent studies are demonstrating the efficacy of the ensemble machine learning approach; hence, the purpose of this study is to assist in the automated detection of fake news. An ensemble approach is adopted to help resolve the identified gap. This study proposed a blended machine learning ensemble model developed from logistic regression, support vector machine, linear discriminant analysis, stochastic gradient descent, and ridge regression, which is then used on a publicly available dataset to predict if a news report is true or not. The proposed model will be appraised with the popular classical machine learning models, while performance metrics such as AUC, ROC, recall, accuracy, precision, and f1-score will be used to measure the performance of the proposed model. Results presented showed that the proposed model outperformed other popular classical machine learning models.

Evaluation of Binding Site Comparison Algorithms and Proteometric Machine Learning Models in the Detection of Protein Pockets Capable of Binding the Same Ligand

2019 ◽  
Author(s):  
Zied Gaieb ◽  
conor parks ◽  
Rommie Amaro
Keyword(s):  
Machine Learning ◽  
Biological Networks ◽  
Protein Targeting ◽  
Performance Metrics ◽  
Protein Fold ◽  
Learning Models ◽  
Protein Pockets ◽  
Comparison Algorithms ◽  
Protein Pocket ◽  
Machine Learning Models

<div> <div> <div> <p>Non linearities of biological networks present ample opportunity for synergistic protein targeting combinations. Yet, to date, our ability to design multi-target inhibitors and predict polypharmacology binding profiles remains limited. Herein, we present a systematic benchmarking of protein pocket comparison algorithms from the literature, as well as novel machine learning models developed to predict whether two proteins will bind the same ligand. The results demonstrate that previously reported performance metrics from the literature could be inflated due to a bias towards proteins of similar folds when identifying protein capable of binding the same ligand. This observation motivated a more in-depth evaluation of the methods against two subsets of same and cross protein fold comparisons. In a head to head comparison using the cross protein fold subset, we found that the proteometric machine learning models were the best performing models overall. </p> </div> </div> </div>

Evaluation of Binding Site Comparison Algorithms and Proteometric Machine Learning Models in the Detection of Protein Pockets Capable of Binding the Same Ligand

2019 ◽  
Author(s):  
Zied Gaieb ◽  
conor parks ◽  
Rommie Amaro
Keyword(s):  
Machine Learning ◽  
Biological Networks ◽  
Protein Targeting ◽  
Performance Metrics ◽  
Protein Fold ◽  
Learning Models ◽  
Protein Pockets ◽  
Comparison Algorithms ◽  
Protein Pocket ◽  
Machine Learning Models

<div> <div> <div> <p>Non linearities of biological networks present ample opportunity for synergistic protein targeting combinations. Yet, to date, our ability to design multi-target inhibitors and predict polypharmacology binding profiles remains limited. Herein, we present a systematic benchmarking of protein pocket comparison algorithms from the literature, as well as novel machine learning models developed to predict whether two proteins will bind the same ligand. The results demonstrate that previously reported performance metrics from the literature could be inflated due to a bias towards proteins of similar folds when identifying protein capable of binding the same ligand. This observation motivated a more in-depth evaluation of the methods against two subsets of same and cross protein fold comparisons. In a head to head comparison using the cross protein fold subset, we found that the proteometric machine learning models were the best performing models overall. </p> </div> </div> </div>

Drug Synergy Prediction Using Dynamic Mutation based Differential Evolution

2020 ◽  
Vol 26 ◽  
Author(s):  
Manjit Kaur ◽  
Dilbag Singh ◽  
Vijay Kumar
Keyword(s):  
Machine Learning ◽  
Differential Evolution ◽  
High Throughput Screening ◽  
Performance Metrics ◽  
Cancer Therapies ◽  
Learning Models ◽  
Drug Synergy ◽  
Dynamic Mutation ◽  
Mixture Density ◽  
Machine Learning Models

Purpose: In cancer therapies, drug combinations have shown significance accuracy and minimal side effects than the single drug administration. Therefore, the drug synergy has drawn great interest from pharmaceutical companies and researchers. Unfortunately, the prediction of drug synergy score by purely investigational means is only possible on small groups of drugs. Methods: With an advancement in high-throughput screening (HTS), the size of drug synergy datasets has grown enormously in recent years. Therefore, recently, many machine learning models have been utilized to predict the drug synergy score. However, the majority of these machine learning models suffer from over-fitting and hyper-parameters tuning issues. Results: Therefore, a novel deep bidirectional mixture density network (BMDN) model is proposed. To optimize the hyperparameters of BMDN, a dynamic mutation based multi-objective differential evolution is used. Extensive experiments are drawn by using the NCI-ALMANAC drug synergy dataset with over 290,000 synergy determinations. Conclusions: Experimental results shows that BMDN outperforms the exisitng drug synergy models in terms of various performance metrics.

scholarly journals Supervised ensemble learning methods towards automatically filtering Urdu fake news within social media

2021 ◽  
Vol 7 ◽  
pp. e425
Author(s):  
Muhammad Pervez Akhter ◽  
Jiangbin Zheng ◽  
Farkhanda Afzal ◽  
Hui Lin ◽  
Saleem Riaz ◽  
...  
Keyword(s):  
Machine Learning ◽  
Ensemble Learning ◽  
English Language ◽  
Performance Metrics ◽  
Area Under The Curve ◽  
Absolute Error ◽  
Fake News ◽  
Learning Models ◽  
Learning Methods ◽  
Machine Learning Models

The popularity of the internet, smartphones, and social networks has contributed to the proliferation of misleading information like fake news and fake reviews on news blogs, online newspapers, and e-commerce applications. Fake news has a worldwide impact and potential to change political scenarios, deceive people into increasing product sales, defaming politicians or celebrities, and misguiding visitors to stop visiting a place or country. Therefore, it is vital to find automatic methods to detect fake news online. In several past studies, the focus was the English language, but the resource-poor languages have been completely ignored because of the scarcity of labeled corpus. In this study, we investigate this issue in the Urdu language. Our contribution is threefold. First, we design an annotated corpus of Urdu news articles for the fake news detection tasks. Second, we explore three individual machine learning models to detect fake news. Third, we use five ensemble learning methods to ensemble the base-predictors’ predictions to improve the fake news detection system’s overall performance. Our experiment results on two Urdu news corpora show the superiority of ensemble models over individual machine learning models. Three performance metrics balanced accuracy, the area under the curve, and mean absolute error used to find that Ensemble Selection and Vote models outperform the other machine learning and ensemble learning models.

scholarly journals COVID-19 Identification using Machine Learning Classifiers with GLCM Features of Chest X-ray Images

2021 ◽  
Vol 18 (23) ◽  
pp. 46
Author(s):  
Sudeep D. Thepade ◽  
Hrishikesh Jha
Keyword(s):  
Machine Learning ◽  
Performance Metrics ◽  
Automated System ◽  
Learning Models ◽  
X Ray ◽  
Chest X Ray ◽  
Rapid Pace ◽  
Occurrence Matrix ◽  
Machine Learning Models

COVID-19 is an ongoing pandemic, and is also known by the name coronavirus. It was originally discovered in Wuhan, China, in December, 2019. Since then, it has been increasing rapidly worldwide. Since it has been increasing at such a rapid pace, testing equipment has limited availability. Also, this disease spreads very quickly, so it is better if it is detected earlier, in order so that it can be stopped from spreading. Therefore, the importance of early detection has increased; however, because of the shortage of testing sets, it is a necessity to develop an automated system that can detect whether the COVID-19 disease is present in a person or not as early as possible. Therefore, in this work, to extract features from X-ray images of the chest, we have made use of the Gray Level Co-occurrence Matrix (GLCM). After extracting these features for the classification of the images, we used different machine learning models, and an ensemble of machine learning models, to classify X-ray images of the chest as COVID-19, Normal, Pneumonia-bac, or Pneumonia-vir. Considering the average of performance metrics, the ensemble of Random Forest-MLP gave the best result among the variations.

Sign in / Sign up

Export Citation Format

Share Document