Ensemble Machine Learning Approaches for Proteogenomic Cancer Studies

Abstract Background: The identification of important proteins is critical for medical diagnosis and prognosis in common diseases. Diverse sets of computational tools were developed for omics data reductions and protein selections. However, standard statistical models with single feature selection involve the multi-testing burden of low power with the available limited samples. Furthermore, high correlations among proteins with high redundancy and moderate effects often lead to unstable selections and cause reproducibility issues. Ensemble feature selection in machine learning may identify a stable set of disease biomarkers that could improve the prediction performance of subsequent classification models, and thereby simplify their interpretability. In this study, we developed a three-stage homogeneous ensemble feature selection approach for both identifying proteins and improving prediction accuracy. This approach was implemented and applied to ovarian cancer proteogenomics data sets: 1) binary putative homologous recombination deficiency positive or negative; and 2) multiple mRNA classes (differentiated, proliferative, immunoreactive, mesenchymal, and unknown). We conducted and compared various machine learning approaches with homogeneous ensemble feature selection including random forest, support vector machine, and neural network for predicting both binary and multiple class outcomes. Various performance criteria including sensitivity, specificity, kappa statistics were used to assess the prediction consistency and accuracy. Results: With the proposed three-stage homogeneous ensemble feature selection approaches, prediction accuracy can be improved with the limited sample through continuously reducing errors and redundancy, i.e. Treebag provided 83% prediction accuracy (85% sensitivity and 81% specificity) for binary ovarian outcomes. For mRNA multi-classes classification, our approach provided even better accuracy with increased sample size. Conclusions: Despite the different prediction accuracies from various models, homogeneous ensemble feature selection proposed identified consistent sets of top ranked important markers out of 9606 proteins linked to the binary disease and multiple mRNA class outcomes.

Download Full-text

Tree-based homogeneous ensemble model with feature selection for diabetic retinopathy prediction

Jurnal Teknologi dan Sistem Komputer ◽

10.14710/jtsiskom.2020.13669 ◽

2020 ◽

Vol 8 (4) ◽

pp. 297-303

Author(s):

Tamunopriye Ene Dagogo-George ◽

Hammed Adeleye Mojeed ◽

Abdulateef Oluwagbemiga Balogun ◽

Modinat Abolore Mabayoje ◽

Shakirat Aderonke Salihu

Keyword(s):

Machine Learning ◽

Feature Selection ◽

Diabetic Retinopathy ◽

Ensemble Methods ◽

Predictive Performance ◽

Ensemble Classification ◽

Support Vector ◽

Learning Approaches ◽

Vector Machines ◽

Homogeneous Ensemble

Diabetic Retinopathy (DR) is a condition that emerges from prolonged diabetes, causing severe damages to the eyes. Early diagnosis of this disease is highly imperative as late diagnosis may be fatal. Existing studies employed machine learning approaches with Support Vector Machines (SVM) having the highest performance on most analyses and Decision Trees (DT) having the lowest. However, SVM has been known to suffer from parameter and kernel selection problems, which undermine its predictive capability. Hence, this study presents homogenous ensemble classification methods with DT as the base classifier to optimize predictive performance. Boosting and Bagging ensemble methods with feature selection were employed, and experiments were carried out using Python Scikit Learn libraries on DR datasets extracted from UCI Machine Learning repository. Experimental results showed that Bagged and Boosted DT were better than SVM. Specifically, Bagged DT performed best with accuracy 65.38 %, f-score 0.664, and AUC 0.731, followed by Boosted DT with accuracy 65.42 %, f-score 0.655, and AUC 0.724 when compared to SVM (accuracy 65.16 %, f-score 0.652, and AUC 0.721). These results indicate that DT's predictive performance can be optimized by employing the homogeneous ensemble methods to outperform SVM in predicting DR.

Download Full-text

Abstract TP458: High Accuracy of Predictive Models for SAH Using Different Machine Learning Approaches

Stroke ◽

10.1161/str.51.suppl_1.tp458 ◽

2020 ◽

Vol 51 (Suppl_1) ◽

Author(s):

Paul Litvak ◽

Jeevan Medikonda ◽

Girish Menon ◽

Pitchaiah Mandava

Keyword(s):

Machine Learning ◽

Predictive Models ◽

Prediction Accuracy ◽

Support Vector ◽

World Federation ◽

Learning Approaches ◽

Flow Models ◽

Multi Stage ◽

Stage 1 ◽

Categorical Scale

Background: Patients suffering from subarachnoid hemorrhage (SAH) have poor long-term outcomes. There are predictive models for ischemic and hemorrhagic stroke. However, there is paucity of models for SAH. Machine learning concepts were applied to build multi-stage Neural Networks (NN), Support Vector Machines (SVM) and Keras/Tensor Flow models to predict SAH outcomes. Methods: A database of ~800 aneurysmal SAH patients from Kasturba Medical College was utilized. Baseline variables of World Federation of Neurosurgeons 5-point scale (WFNS 1-5), age, gender, and presence/absence of hypertension and diabetes were considered in Stage 1. Stage 2 included all Stage 1 variables along with presence/absence of radiologic signs vasospasm and ischemia. Stage 3 includes earlier 2 stages and discharge Glasgow Outcome Scale (GOS 1-5). GOS at 3 months was predicted using 2-layer NN/SVM/Keras-TensorFlow models on the five point categorical scale as well as dichotomized to dead/alive and favorable (GOS 4-5) or unfavorable (GOS 1-3). Prediction accuracy of models was compared to the recorded GOS. Results: Prediction accuracy shown as percentages (See Table) for all three stages was similar for SVM, NN and Keras/TensorFlow models. Accuracy was remarkably higher with dichotomization compared to the complete five point GOS categorical scale. Conclusions: SVM, NN, and Keras-TensorFlow based machine learning models can be used to predict SAH outcomes to a high degree of accuracy. These powerful predictive models can be used to prognosticate and select patients into trials.

Download Full-text

Computational prediction of implantation outcome after embryo transfer

Health Informatics Journal ◽

10.1177/1460458219892138 ◽

2019 ◽

Vol 26 (3) ◽

pp. 1810-1826 ◽

Cited By ~ 3

Author(s):

Behnaz Raef ◽

Masoud Maleki ◽

Reza Ferdousi

Keyword(s):

Machine Learning ◽

Feature Selection ◽

Prediction Model ◽

Embryo Transfer ◽

Area Under The Curve ◽

Computational Prediction ◽

Support Vector ◽

Human Menopausal Gonadotropin ◽

Optimum Number ◽

Learning Approaches

The aim of this study is to develop a computational prediction model for implantation outcome after an embryo transfer cycle. In this study, information of 500 patients and 1360 transferred embryos, including cleavage and blastocyst stages and fresh or frozen embryos, from April 2016 to February 2018, were collected. The dataset containing 82 attributes and a target label (indicating positive and negative implantation outcomes) was constructed. Six dominant machine learning approaches were examined based on their performance to predict embryo transfer outcomes. Also, feature selection procedures were used to identify effective predictive factors and recruited to determine the optimum number of features based on classifiers performance. The results revealed that random forest was the best classifier (accuracy = 90.40% and area under the curve = 93.74%) with optimum features based on a 10-fold cross-validation test. According to the Support Vector Machine-Feature Selection algorithm, the ideal numbers of features are 78. Follicle stimulating hormone/human menopausal gonadotropin dosage for ovarian stimulation was the most important predictive factor across all examined embryo transfer features. The proposed machine learning-based prediction model could predict embryo transfer outcome and implantation of embryos with high accuracy, before the start of an embryo transfer cycle.

Download Full-text

Evaluation of Machine Learning Approaches to Predict Soil Organic Matter and pH Using vis-NIR Spectra

Sensors ◽

10.3390/s19020263 ◽

2019 ◽

Vol 19 (2) ◽

pp. 263 ◽

Cited By ~ 16

Author(s):

Meihua Yang ◽

Dongyun Xu ◽

Songchao Chen ◽

Hongyi Li ◽

Zhou Shi

Keyword(s):

Machine Learning ◽

Organic Matter ◽

Soil Organic Matter ◽

Least Squares ◽

Paddy Soil ◽

Prediction Accuracy ◽

Accurate Determination ◽

Support Vector ◽

Learning Approaches ◽

Lower Yangtze

Soil organic matter (SOM) and pH are essential soil fertility indictors of paddy soil in the middle-lower Yangtze Plain. Rapid, non-destructive and accurate determination of SOM and pH is vital to preventing soil degradation caused by inappropriate land management practices. Visible-near infrared (vis-NIR) spectroscopy with multivariate calibration can be used to effectively estimate soil properties. In this study, 523 soil samples were collected from paddy fields in the Yangtze Plain, China. Four machine learning approaches—partial least squares regression (PLSR), least squares-support vector machines (LS-SVM), extreme learning machines (ELM) and the Cubist regression model (Cubist)—were used to compare the prediction accuracy based on vis-NIR full bands and bands reduced using the genetic algorithm (GA). The coefficient of determination (R2), root mean square error (RMSE), and ratio of performance to inter-quartile distance (RPIQ) were used to assess the prediction accuracy. The ELM with GA reduced bands was the best model for SOM (SOM: R2 = 0.81, RMSE = 5.17, RPIQ = 2.87) and pH (R2 = 0.76, RMSE = 0.43, RPIQ = 2.15). The performance of the LS-SVM for pH prediction did not differ significantly between the model with GA (R2 = 0.75, RMSE = 0.44, RPIQ = 2.08) and without GA (R2 = 0.74, RMSE = 0.45, RPIQ = 2.07). Although a slight increase was observed when ELM were used for prediction of SOM and pH using reduced bands (SOM: R2 = 0.81, RMSE = 5.17, RPIQ = 2.87; pH: R2 = 0.76, RMSE = 0.43, RPIQ = 2.15) compared with full bands (R2 = 0.81, RMSE = 5.18, RPIQ = 2.83; pH: R2 = 0.76, RMSE = 0.45, RPIQ = 2.07), the number of wavelengths was greatly reduced (SOM: 201 to 44; pH: 201 to 32). Thus, the ELM coupled with reduced bands by GA is recommended for prediction of properties of paddy soil (SOM and pH) in the middle-lower Yangtze Plain.

Download Full-text

Diagnostic Performance of 2D and 3D T2WI-Based Radiomics Features With Machine Learning Algorithms to Distinguish Solid Solitary Pulmonary Lesion

Frontiers in Oncology ◽

10.3389/fonc.2021.683587 ◽

2021 ◽

Vol 11 ◽

Author(s):

Qi Wan ◽

Jiaxuan Zhou ◽

Xiaoying Xia ◽

Jianfeng Hu ◽

Peng Wang ◽

...

Keyword(s):

Machine Learning ◽

Feature Selection ◽

Diagnostic Performance ◽

Feature Selection Method ◽

Machine Learning Algorithms ◽

Support Vector ◽

Learning Approaches ◽

Selection Methods ◽

Linear Discriminant ◽

2D And 3D

ObjectiveTo evaluate the performance of 2D and 3D radiomics features with different machine learning approaches to classify SPLs based on magnetic resonance(MR) T2 weighted imaging (T2WI).Material and MethodsA total of 132 patients with pathologically confirmed SPLs were examined and randomly divided into training (n = 92) and test datasets (n = 40). A total of 1692 3D and 1231 2D radiomics features per patient were extracted. Both radiomics features and clinical data were evaluated. A total of 1260 classification models, comprising 3 normalization methods, 2 dimension reduction algorithms, 3 feature selection methods, and 10 classifiers with 7 different feature numbers (confined to 3–9), were compared. The ten-fold cross-validation on the training dataset was applied to choose the candidate final model. The area under the receiver operating characteristic curve (AUC), precision-recall plot, and Matthews Correlation Coefficient were used to evaluate the performance of machine learning approaches.ResultsThe 3D features were significantly superior to 2D features, showing much more machine learning combinations with AUC greater than 0.7 in both validation and test groups (129 vs. 11). The feature selection method Analysis of Variance(ANOVA), Recursive Feature Elimination(RFE) and the classifier Logistic Regression(LR), Linear Discriminant Analysis(LDA), Support Vector Machine(SVM), Gaussian Process(GP) had relatively better performance. The best performance of 3D radiomics features in the test dataset (AUC = 0.824, AUC-PR = 0.927, MCC = 0.514) was higher than that of 2D features (AUC = 0.740, AUC-PR = 0.846, MCC = 0.404). The joint 3D and 2D features (AUC=0.813, AUC-PR = 0.926, MCC = 0.563) showed similar results as 3D features. Incorporating clinical features with 3D and 2D radiomics features slightly improved the AUC to 0.836 (AUC-PR = 0.918, MCC = 0.620) and 0.780 (AUC-PR = 0.900, MCC = 0.574), respectively.ConclusionsAfter algorithm optimization, 2D feature-based radiomics models yield favorable results in differentiating malignant and benign SPLs, but 3D features are still preferred because of the availability of more machine learning algorithmic combinations with better performance. Feature selection methods ANOVA and RFE, and classifier LR, LDA, SVM and GP are more likely to demonstrate better diagnostic performance for 3D features in the current study.

Download Full-text

Detecting Web-Based Attacks with SHAP and Tree Ensemble Machine Learning Methods

Applied Sciences ◽

10.3390/app12010060 ◽

2021 ◽

Vol 12 (1) ◽

pp. 60

Author(s):

Samuel Ndichu ◽

Sangwook Kim ◽

Seiichi Ozawa ◽

Tao Ban ◽

Takeshi Takahashi ◽

...

Keyword(s):

Machine Learning ◽

Feature Selection ◽

Ensemble Methods ◽

Learning Approaches ◽

Rule Mining ◽

Web Based ◽

Tree Form ◽

Domain Experts ◽

Domain Names ◽

Ensemble Machine Learning

Attacks using Uniform Resource Locators (URLs) and their JavaScript (JS) code content to perpetrate malicious activities on the Internet are rampant and continuously evolving. Methods such as blocklisting, client honeypots, domain reputation inspection, and heuristic and signature-based systems are used to detect these malicious activities. Recently, machine learning approaches have been proposed; however, challenges still exist. First, blocklist systems are easily evaded by new URLs and JS code content, obfuscation, fast-flux, cloaking, and URL shortening. Second, heuristic and signature-based systems do not generalize well to zero-day attacks. Third, the Domain Name System allows cybercriminals to easily migrate their malicious servers to hide their Internet protocol addresses behind domain names. Finally, crafting fully representative features is challenging, even for domain experts. This study proposes a feature selection and classification approach for malicious JS code content using Shapley additive explanations and tree ensemble methods. The JS code features are obtained from the Abstract Syntax Tree form of the JS code, sample JS attack codes, and association rule mining. The malicious and benign JS code datasets obtained from Hynek Petrak and the Majestic Million Service were used for performance evaluation. We compared the performance of the proposed method to those of other feature selection methods in the task of malicious JS code content detection. With a recall of 0.9989, our experimental results show that the proposed approach is a better prediction model.

Download Full-text

The Effectiveness of Feature Selection Method in Solar Power Prediction

Journal of Renewable Energy ◽

10.1155/2013/952613 ◽

2013 ◽

Vol 2013 ◽

pp. 1-9 ◽

Cited By ~ 4

Author(s):

Md Rahat Hossain ◽

Amanullah Maung Than Oo ◽

A. B. M. Shawkat Ali

Keyword(s):

Machine Learning ◽

Feature Selection ◽

Prediction Accuracy ◽

Solar Power ◽

Feature Subset Selection ◽

Machine Learning Techniques ◽

Support Vector ◽

Selection Methods ◽

Power Prediction ◽

Learning Techniques

This paper empirically shows that the effect of applying selected feature subsets on machine learning techniques significantly improves the accuracy for solar power prediction. Experiments are performed using five well-known wrapper feature selection methods to obtain the solar power prediction accuracy of machine learning techniques with selected feature subsets. For all the experiments, the machine learning techniques, namely, least median square (LMS), multilayer perceptron (MLP), and support vector machine (SVM), are used. Afterwards, these results are compared with the solar power prediction accuracy of those same machine leaning techniques (i.e., LMS, MLP, and SVM) but without applying feature selection methods (WAFS). Experiments are carried out using reliable and real life historical meteorological data. The comparison between the results clearly shows that LMS, MLP, and SVM provide better prediction accuracy (i.e., reduced MAE and MASE) with selected feature subsets than without selected feature subsets. Experimental results of this paper facilitate to make a concrete verdict that providing more attention and effort towards the feature subset selection aspect (e.g., selected feature subsets on prediction accuracy which is investigated in this paper) can significantly contribute to improve the accuracy of solar power prediction.

Download Full-text

Using Machine Learning Approaches to Predict Target Gene Expression in Rice T-DNA Insertional Mutants

Frontiers in Genetics ◽

10.3389/fgene.2021.798107 ◽

2021 ◽

Vol 12 ◽

Author(s):

Ching-Hsuan Chien ◽

Lan-Ying Huang ◽

Shuen-Fang Lo ◽

Liang-Jwu Chen ◽

Chi-Chou Liao ◽

...

Keyword(s):

Gene Expression ◽

Machine Learning ◽

Feature Selection ◽

Target Genes ◽

Promoter Sequence ◽

Activation Mechanism ◽

Support Vector ◽

Regulatory Sequence ◽

Learning Approaches ◽

Local Sequence

To change the expression of the flanking genes by inserting T-DNA into the genome is commonly used in rice functional gene research. However, whether the expression of a gene of interest is enhanced must be validated experimentally. Consequently, to improve the efficiency of screening activated genes, we established a model to predict gene expression in T-DNA mutants through machine learning methods. We gathered experimental datasets consisting of gene expression data in T-DNA mutants and captured the PROMOTER and MIDDLE sequences for encoding. In first-layer models, support vector machine (SVM) models were constructed with nine features consisting of information about biological function and local and global sequences. Feature encoding based on the PROMOTER sequence was weighted by logistic regression. The second-layer models integrated 16 first-layer models with minimum redundancy maximum relevance (mRMR) feature selection and the LADTree algorithm, which were selected from nine feature selection methods and 65 classified methods, respectively. The accuracy of the final two-layer machine learning model, referred to as TIMgo, was 99.3% based on fivefold cross-validation, and 85.6% based on independent testing. We discovered that the information within the local sequence had a greater contribution than the global sequence with respect to classification. TIMgo had a good predictive ability for target genes within 20 kb from the 35S enhancer. Based on the analysis of significant sequences, the G-box regulatory sequence may also play an important role in the activation mechanism of the 35S enhancer.

Download Full-text

Ensemble Machine Learning Approach Improves Predicted Spatial Variation of Surface Soil Organic Carbon Stocks in Data-Limited Northern Circumpolar Region

Frontiers in Big Data ◽

10.3389/fdata.2020.528441 ◽

2020 ◽

Vol 3 ◽

Author(s):

Umakant Mishra ◽

Sagar Gautam ◽

William J. Riley ◽

Forrest M. Hoffman

Keyword(s):

Machine Learning ◽

Environmental Factors ◽

Soil Properties ◽

Spatial Variation ◽

Prediction Accuracy ◽

Gradient Boosting ◽

Support Vector ◽

Learning Approaches ◽

Regression Kriging ◽

Soc Stocks

Various approaches of differing mathematical complexities are being applied for spatial prediction of soil properties. Regression kriging is a widely used hybrid approach of spatial variation that combines correlation between soil properties and environmental factors with spatial autocorrelation between soil observations. In this study, we compared four machine learning approaches (gradient boosting machine, multinarrative adaptive regression spline, random forest, and support vector machine) with regression kriging to predict the spatial variation of surface (0–30 cm) soil organic carbon (SOC) stocks at 250-m spatial resolution across the northern circumpolar permafrost region. We combined 2,374 soil profile observations (calibration datasets) with georeferenced datasets of environmental factors (climate, topography, land cover, bedrock geology, and soil types) to predict the spatial variation of surface SOC stocks. We evaluated the prediction accuracy at randomly selected sites (validation datasets) across the study area. We found that different techniques inferred different numbers of environmental factors and their relative importance for prediction of SOC stocks. Regression kriging produced lower prediction errors in comparison to multinarrative adaptive regression spline and support vector machine, and comparable prediction accuracy to gradient boosting machine and random forest. However, the ensemble median prediction of SOC stocks obtained from all four machine learning techniques showed highest prediction accuracy. Although the use of different approaches in spatial prediction of soil properties will depend on the availability of soil and environmental datasets and computational resources, we conclude that the ensemble median prediction obtained from multiple machine learning approaches provides greater spatial details and produces the highest prediction accuracy. Thus an ensemble prediction approach can be a better choice than any single prediction technique for predicting the spatial variation of SOC stocks.

Download Full-text

Failure Prediction of Aircraft Equipment Using Machine Learning with a Hybrid Data Preparation Method

Scientific Programming ◽

10.1155/2020/8616039 ◽

2020 ◽

Vol 2020 ◽

pp. 1-10

Author(s):

Kadir Celikmih ◽

Onur Inan ◽

Harun Uguz

Keyword(s):

Machine Learning ◽

Feature Selection ◽

Feature Selection Method ◽

Absolute Error ◽

Machine Learning Algorithms ◽

Performance Criteria ◽

Aviation Industry ◽

Support Vector ◽

Data Preparation ◽

Hybrid Data

There is a large amount of information and maintenance data in the aviation industry that could be used to obtain meaningful results in forecasting future actions. This study aims to introduce machine learning models based on feature selection and data elimination to predict failures of aircraft systems. Maintenance and failure data for aircraft equipment across a period of two years were collected, and nine input and one output variables were meticulously identified. A hybrid data preparation model is proposed to improve the success of failure count prediction in two stages. In the first stage, ReliefF, a feature selection method for attribute evaluation, is used to find the most effective and ineffective parameters. In the second stage, a K-means algorithm is modified to eliminate noisy or inconsistent data. Performance of the hybrid data preparation model on the maintenance dataset of the equipment is evaluated by Multilayer Perceptron (MLP) as Artificial Neural network (ANN), Support Vector Regression (SVR), and Linear Regression (LR) as machine learning algorithms. Moreover, performance criteria such as the Correlation Coefficient (CC), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE) are used to evaluate the models. The results indicate that the hybrid data preparation model is successful in predicting the failure count of the equipment.

Download Full-text