scholarly journals A consistent evaluation of miRNA-disease association prediction models

2020 ◽  
Author(s):  
Ngan Thi Dong ◽  
Megha Khosla
Keyword(s):  
Machine Learning ◽  
Model Evaluation ◽  
Performance Metrics ◽  
Prediction Models ◽  
Disease Association ◽  
Evaluation Framework ◽  
Average Precision ◽  
Disease Similarity ◽  
Crucial Problem ◽  
Set Up

AbstractMotivationA variety of machine learning based approaches have been applied to predicting miRNA-disease association. Although promising, the evaluation set up to measure prediction performance is inconsistent making it difficult to assess the actual progress. A more acute problem is that most of the models overlook the problem of data leakage due to the use of precomputed miRNA and disease similarity features.ResultsWe unearth a crucial problem of data leakage in evaluation of machine learning models for miRNA-disease association prediction. In particular, information from test set, in the form of precomputed input features for miRNA and disease, is used during training of the model. Moreover, we point out problems in the widely used performance metrics used in model evaluation. While resolving the issues of data leakage and model evaluation, we perform an indepth study of 3 recent models along with our proposed 9 variants of these models. Our proposed variants have resulted in improvements in Average Precision scores (as compared to original models) by approximately 287.7% and 36.7% on HMDDv2.0 (AP:0.504) and HMDDv3.0 (AP: 0.216) datasets respectively.Availability and ImplementationWe release a unified evaluation framework including all models and datasets at https://git.l3s.uni-hannover.de/dong/simplifying_mirna_disease.

scholarly journals Machine Learning Methods Applied to the Prediction of Pseudo-nitzschia spp. Blooms in the Galician Rias Baixas (NW Spain)

2021 ◽  
Vol 10 (4) ◽  
pp. 199
Author(s):  
Francisco M. Bellas Aláez ◽  
Jesus M. Torres Palenzuela ◽  
Evangelos Spyrakos ◽  
Luis González Vilas
Keyword(s):  
Machine Learning ◽  
Performance Metrics ◽  
Prediction Models ◽  
Support Vector ◽  
False Alarms ◽  
Learning Approaches ◽  
Learning Methods ◽  
Machine Learning Methods ◽  
Rías Baixas ◽  
New Algorithms

This work presents new prediction models based on recent developments in machine learning methods, such as Random Forest (RF) and AdaBoost, and compares them with more classical approaches, i.e., support vector machines (SVMs) and neural networks (NNs). The models predict Pseudo-nitzschia spp. blooms in the Galician Rias Baixas. This work builds on a previous study by the authors (doi.org/10.1016/j.pocean.2014.03.003) but uses an extended database (from 2002 to 2012) and new algorithms. Our results show that RF and AdaBoost provide better prediction results compared to SVMs and NNs, as they show improved performance metrics and a better balance between sensitivity and specificity. Classical machine learning approaches show higher sensitivities, but at a cost of lower specificity and higher percentages of false alarms (lower precision). These results seem to indicate a greater adaptation of new algorithms (RF and AdaBoost) to unbalanced datasets. Our models could be operationally implemented to establish a short-term prediction system.

Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning

2021 ◽  
Author(s):  
Chenxi Huang ◽  
Shu-Xia Li ◽  
César Caraballo ◽  
Frederick A. Masoudi ◽  
John S. Rumsfeld ◽  
...  
Keyword(s):  
Machine Learning ◽  
Risk Prediction ◽  
Health Care Professionals ◽  
Clinical Decision Making ◽  
Performance Metrics ◽  
Prediction Models ◽  
Learning Models ◽  
Risk Prediction Models ◽  
Clinical Risk ◽  
Machine Learning Models

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics. Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics. Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.

Predictive modelling of hospital readmission: Evaluation of different preprocessing techniques on machine learning classifiers

2021 ◽  
Vol 25 (5) ◽  
pp. 1073-1098
Author(s):  
Nor Hamizah Miswan ◽  
Chee Seng Chan ◽  
Chong Guan Ng
Keyword(s):  
Machine Learning ◽  
Hospital Readmission ◽  
Performance Metrics ◽  
Predictive Performance ◽  
Predictive Modelling ◽  
Machine Learning Classifiers ◽  
Learning Classifiers ◽  
Set Up ◽  
The Right ◽  
The Impact

Hospital readmission is a major cost for healthcare systems worldwide. If patients with a higher potential of readmission could be identified at the start, existing resources could be used more efficiently, and appropriate plans could be implemented to reduce the risk of readmission. Therefore, it is important to predict the right target patients. Medical data is usually noisy, incomplete, and inconsistent. Hence, before developing a prediction model, it is crucial to efficiently set up the predictive model so that improved predictive performance is achieved. The current study aims to analyse the impact of different preprocessing methods on the performance of different machine learning classifiers. The preprocessing applied by previous hospital readmission studies were compared, and the most common approaches highlighted such as missing value imputation, feature selection, data balancing, and feature scaling. The hyperparameters were selected using Bayesian optimisation. The different preprocessing pipelines were assessed using various performance metrics and computational costs. The results indicated that the preprocessing approaches helped improve the model’s prediction of hospital readmission.

Machine Learning for Emergency Department Management

2019 ◽  
Vol 11 (3) ◽  
pp. 19-36 ◽  
Author(s):  
Sofia Benbelkacem ◽  
Farid Kadri ◽  
Baghdad Atmani ◽  
Sondès Chaabane
Keyword(s):  
Machine Learning ◽  
Emergency Department ◽  
Length Of Stay ◽  
Prediction Models ◽  
Pediatric Emergency ◽  
Machine Learning Techniques ◽  
Learning Techniques ◽  
Emergency Department Management ◽  
Increasing Demand ◽  
Set Up

Nowadays, emergency department services are confronted to an increasing demand. This situation causes emergency department overcrowding which often increases the length of stay of patients and leads to strain situations. To overcome this issue, emergency department managers must predict the length of stay. In this work, the researchers propose to use machine learning techniques to set up a methodology that supports the management of emergency departments (EDs). The target of this work is to predict the length of stay of patients in the ED in order to prevent strain situations. The experiments were carried out on a real database collected from the pediatric emergency department (PED) in Lille regional hospital center, France. Different machine learning techniques have been used to build the best prediction models. The results seem better with Naive Bayes, C4.5 and SVM methods. In addition, the models based on a subset of attributes proved to be more efficient than models based on the set of attributes.

scholarly journals A Systematic Methodology to Evaluate Prediction Models for Driving Style Classification

Sensors ◽  
2020 ◽  
Vol 20 (6) ◽  
pp. 1692 ◽  
Author(s):  
Iván Silva ◽  
José Eugenio Naranjo
Keyword(s):  
Machine Learning ◽  
Nearest Neighbor ◽  
Performance Metrics ◽  
Prediction Models ◽  
Statistical Tests ◽  
Area Under The Curve ◽  
The Other ◽  
Support Vector ◽  
Classification Models ◽  
K Nearest Neighbor

Identifying driving styles using classification models with in-vehicle data can provide automated feedback to drivers on their driving behavior, particularly if they are driving safely. Although several classification models have been developed for this purpose, there is no consensus on which classifier performs better at identifying driving styles. Therefore, more research is needed to evaluate classification models by comparing performance metrics. In this paper, a data-driven machine-learning methodology for classifying driving styles is introduced. This methodology is grounded in well-established machine-learning (ML) methods and literature related to driving-styles research. The methodology is illustrated through a study involving data collected from 50 drivers from two different cities in a naturalistic setting. Five features were extracted from the raw data. Fifteen experts were involved in the data labeling to derive the ground truth of the dataset. The dataset fed five different models (Support Vector Machines (SVM), Artificial Neural Networks (ANN), fuzzy logic, k-Nearest Neighbor (kNN), and Random Forests (RF)). These models were evaluated in terms of a set of performance metrics and statistical tests. The experimental results from performance metrics showed that SVM outperformed the other four models, achieving an average accuracy of 0.96, F1-Score of 0.9595, Area Under the Curve (AUC) of 0.9730, and Kappa of 0.9375. In addition, Wilcoxon tests indicated that ANN predicts differently to the other four models. These promising results demonstrate that the proposed methodology may support researchers in making informed decisions about which ML model performs better for driving-styles classification.

scholarly journals Towards improved and more routine Earth system model evaluation in CMIP

2016 ◽  
Vol 7 (4) ◽  
pp. 813-830 ◽  
Author(s):  
Veronika Eyring ◽  
Peter J. Gleckler ◽  
Christoph Heinze ◽  
Ronald J. Stouffer ◽  
Karl E. Taylor ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Performance Metrics ◽  
Evaluation Framework ◽  
Systematic Evaluation ◽  
Earth System ◽  
Model Errors ◽  
Model Intercomparison ◽  
Evaluation Tools ◽  
Uncertainty Estimates ◽  
Wide Range

Abstract. The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. More research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well established and considered an essential part of systematic evaluation but have been implemented ad hoc with little community coordination. Given the diversity and complexity of ESM analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently and consistently. Here, we provide a perspective and viewpoint on how a more systematic, open, and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP can be achieved, and announce our intention to implement such a system for CMIP6. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop and apply evaluation tools that are based on the latest scientific knowledge and observational reference, are well suited for routine use, and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output and documentation accessible via the ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for model intercomparison projects using the same data structure and organization as the ESM output. This largely facilitates routine evaluation of the ESMs, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. At the same time, we encourage the community to contribute observations and reanalyses for model evaluation to the obs4MIPs and ana4MIPs archives. The intention is to produce through the ESGF a widely accepted quasi-operational evaluation framework for CMIP6 that would routinely execute a series of standardized evaluation tasks. Over time, as this capability matures, we expect to produce an increasingly systematic characterization of models which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also reveal whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. This framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community.

scholarly journals Towards improved and more routine Earth system model evaluation in CMIP

2016 ◽  
Author(s):  
Veronika Eyring ◽  
Peter J. Gleckler ◽  
Christoph Heinze ◽  
Ronald J. Stouffer ◽  
Karl E. Taylor ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Performance Metrics ◽  
Evaluation Framework ◽  
Systematic Evaluation ◽  
Earth System ◽  
Model Errors ◽  
Model Intercomparison ◽  
Evaluation Tools ◽  
Uncertainty Estimates ◽  
Wide Range

Abstract. The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. Scientifically more research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well-established and considered an essential part of systematic evaluation but are currently implemented ad hoc with little community coordination. Given the diversity and complexity of ESM model analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently to enable a systematic, open and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop evaluation tools that are well suited for routine use and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output accessible via ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for Model Intercomparison Projects using the same data structure and organization. This largely facilitates routine evaluation of the models, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. At the same time, we encourage the community to contribute observations for model evaluation to the obs4MIPs archive. The intention is to produce through ESGF a widely accepted quasi-operational evaluation framework for climate models that would routinely execute a series of standardized evaluation tasks. Over time, as the capability matures, we expect to produce an increasingly systematic characterization of models, which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also expose whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. This framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community.

scholarly journals Machine Learning Prediction Model for Inflammatory Bowel Disease Based on Laboratory Markers. Working Model in a Discovery Cohort Study

2021 ◽  
Vol 10 (20) ◽  
pp. 4745
Author(s):  
Sebastian Kraszewski ◽  
Witold Szczurek ◽  
Julia Szymczak ◽  
Monika Reguła ◽  
Katarzyna Neubauer
Keyword(s):  
Machine Learning ◽  
Inflammatory Bowel Disease ◽  
Prediction Model ◽  
Bowel Disease ◽  
Prediction Models ◽  
Wide Spectrum ◽  
Clinical Problem ◽  
Disease Diagnosis ◽  
Average Precision ◽  
Inflammatory Bowel

Inflammatory bowel disease (IBD) is a chronic, incurable disease involving the gastrointestinal tract. It is characterized by complex, unclear pathogenesis, increased prevalence worldwide, and a wide spectrum of extraintestinal manifestations and comorbidities. Recognition of IBD remains challenging and delays in disease diagnosis still poses a significant clinical problem as it negatively impacts disease outcome. The main diagnostic tool in IBD continues to be invasive endoscopy. We aimed to create an IBD machine learning prediction model based on routinely performed blood, urine, and fecal tests. Based on historical patients’ data (702 medical records: 319 records from 180 patients with ulcerative colitis (UC) and 383 records from 192 patients with Crohn’s disease (CD)), and using a few simple machine learning classificators, we optimized necessary hyperparameters in order to get reliable few-features prediction models separately for CD and UC. Most robust classificators belonging to the random forest family obtained 97% and 91% mean average precision for CD and UC, respectively. For comparison, the commonly used one-parameter approach based on the C-reactive protein (CRP) level demonstrated only 81% and 61% average precision for CD and UC, respectively. Results of our study suggest that machine learning prediction models based on basic blood, urine, and fecal markers may with high accuracy support the diagnosis of IBD. However, the test requires validation in a prospective cohort.

scholarly journals Predicting Drug-Disease Association Based on Ensemble Strategy

2021 ◽  
Vol 12 ◽  
Author(s):  
Jianlin Wang ◽  
Wenxiu Wang ◽  
Chaokun Yan ◽  
Junwei Luo ◽  
Ge Zhang
Keyword(s):  
Prediction Models ◽  
Drug Repositioning ◽  
Predictive Ability ◽  
Disease Association ◽  
New Drugs ◽  
Similarity Network ◽  
Disease Similarity ◽  
Ensemble Strategy ◽  
Disease Associations ◽  
Reducing Costs

Drug repositioning is used to find new uses for existing drugs, effectively shortening the drug research and development cycle and reducing costs and risks. A new model of drug repositioning based on ensemble learning is proposed. This work develops a novel computational drug repositioning approach called CMAF to discover potential drug-disease associations. First, for new drugs and diseases or unknown drug-disease pairs, based on their known neighbor information, an association probability can be obtained by implementing the weighted K nearest known neighbors (WKNKN) method and improving the drug-disease association information. Then, a new drug similarity network and new disease similarity network can be constructed. Three prediction models are applied and ensembled to enable the final association of drug-disease pairs based on improved drug-disease association information and the constructed similarity network. The experimental results demonstrate that the developed approach outperforms recent state-of-the-art prediction models. Case studies further confirm the predictive ability of the proposed method. Our proposed method can effectively improve the prediction results.

scholarly journals Machine Learning Prediction Models for Chronic Kidney Disease using National Health Insurance Claim Data in Taiwan

2020 ◽  
Author(s):  
Surya Krishnamurthy ◽  
Kapeleshh KS ◽  
Erik Dovgan ◽  
Mitja Luštrek ◽  
Barbara Gradišek Piletič ◽  
...  
Keyword(s):  
Machine Learning ◽  
Chronic Kidney Disease ◽  
Health Insurance ◽  
Kidney Disease ◽  
National Health Insurance ◽  
National Health ◽  
Performance Metrics ◽  
Prediction Models ◽  
Research Database ◽  
Number Of Patients

ABSTRACTBackground and ObjectiveChronic kidney disease (CKD) represent a heavy burden on the healthcare system because of the increasing number of patients, high risk of progression to end-stage renal disease, and poor prognosis of morbidity and mortality. The aim of this study is to develop a machine-learning model that uses the comorbidity and medication data, obtained from Taiwan’s National Health Insurance Research Database, to forecast whether an individual will develop CKD within the next 6 or 12 months, and thus forecast the prevalence in the population.MethodsA total of 18,000 people with CKD and 72,000 people without CKD diagnosis along with the past two years of medication and comorbidity data matched by propensity score were used to build a predicting model. A series of approaches were tested, including Convoluted Neural Networks (CNN). 5-fold cross-validation was used to assess the performance metrics of the algorithms.ResultsBoth for the 6 month and 12-month models, the CNN approach performed best, with the AUROC of 0.957 and 0.954, respectively. The most prominent features in the tree-based models were identified, including diabetes mellitus, age, gout, and medications such as sulfonamides, angiotensins which had an impact on the progression of CKD.ConclusionsThe model proposed in this study can be a useful tool for the policy-makers helping them in predicting the trends of CKD in the population in the next 6 to 12 months. Information provided by this model can allow closely monitoring the people with risk, early detection of CKD, better allocation of resources, and patient-centric management

Sign in / Sign up

Export Citation Format

Share Document