Improving Reliability Estimation for Individual Numeric Predictions: A Machine Learning Approach

2021 ◽  
Author(s):  
Gediminas Adomavicius ◽  
Yaqiong Wang
Keyword(s):  
Machine Learning ◽  
General Purpose ◽  
Reliability Estimation ◽  
Machine Learning Techniques ◽  
Data Sets ◽  
Real World Data ◽  
Learning Techniques ◽  
Reliability Indicator ◽  
Machine Learning Approach ◽  
Prediction Reliability

Numerical predictive modeling is widely used in different application domains. Although many modeling techniques have been proposed, and a number of different aggregate accuracy metrics exist for evaluating the overall performance of predictive models, other important aspects, such as the reliability (or confidence and uncertainty) of individual predictions, have been underexplored. We propose to use estimated absolute prediction error as the indicator of individual prediction reliability, which has the benefits of being intuitive and providing highly interpretable information to decision makers, as well as allowing for more precise evaluation of reliability estimation quality. As importantly, the proposed reliability indicator allows the reframing of reliability estimation itself as a canonical numeric prediction problem, which makes the proposed approach general-purpose (i.e., it can work in conjunction with any outcome prediction model), alleviates the need for distributional assumptions, and enables the use of advanced, state-of-the-art machine learning techniques to learn individual prediction reliability patterns directly from data. Extensive experimental results on multiple real-world data sets show that the proposed machine learning-based approach can significantly improve individual prediction reliability estimation as compared with a number of baselines from prior work, especially in more complex predictive scenarios.

scholarly journals Malicious web domain identification using online credibility and performance data by considering the class imbalance issue

2019 ◽  
Vol 119 (3) ◽  
pp. 676-696 ◽  
Author(s):  
Zhongyi Hu ◽  
Raymond Chiong ◽  
Ilung Pranata ◽  
Yukun Bao ◽  
Yuqing Lin
Keyword(s):  
Machine Learning ◽  
Class Imbalance ◽  
Performance Data ◽  
Machine Learning Techniques ◽  
Data Sets ◽  
Real World Data ◽  
Content Type ◽  
Domain Identification ◽  
Learning Techniques ◽  
And Performance

Purpose Malicious web domain identification is of significant importance to the security protection of internet users. With online credibility and performance data, the purpose of this paper to investigate the use of machine learning techniques for malicious web domain identification by considering the class imbalance issue (i.e. there are more benign web domains than malicious ones). Design/methodology/approach The authors propose an integrated resampling approach to handle class imbalance by combining the synthetic minority oversampling technique (SMOTE) and particle swarm optimisation (PSO), a population-based meta-heuristic algorithm. The authors use the SMOTE for oversampling and PSO for undersampling. Findings By applying eight well-known machine learning classifiers, the proposed integrated resampling approach is comprehensively examined using several imbalanced web domain data sets with different imbalance ratios. Compared to five other well-known resampling approaches, experimental results confirm that the proposed approach is highly effective. Practical implications This study not only inspires the practical use of online credibility and performance data for identifying malicious web domains but also provides an effective resampling approach for handling the class imbalance issue in the area of malicious web domain identification. Originality/value Online credibility and performance data are applied to build malicious web domain identification models using machine learning techniques. An integrated resampling approach is proposed to address the class imbalance issue. The performance of the proposed approach is confirmed based on real-world data sets with different imbalance ratios.

Machine Learning for Clinical Data Processing

2011 ◽  
pp. 193-215
Author(s):  
Guo-Zheng Li
Keyword(s):  
Machine Learning ◽  
Data Processing ◽  
Clinical Data ◽  
Machine Learning Techniques ◽  
Support Vector ◽  
Data Sets ◽  
The Novel ◽  
Real World Data ◽  
Data Set ◽  
Learning Techniques

This chapter introduces great challenges and the novel machine learning techniques employed in clinical data processing. It argues that the novel machine learning techniques including support vector machines, ensemble learning, feature selection, feature reuse by using multi-task learning, and multi-label learning provide potentially more substantive solutions for decision support and clinical data analysis. The authors demonstrate the generalization performance of the novel machine learning techniques on real world data sets including one data set of brain glioma, one data set of coronary heart disease in Chinese Medicine and some tumor data sets of microarray. More and more machine learning techniques will be developed to improve analysis precision of clinical data sets.

Machine Learning for Clinical Data Processing

2012 ◽  
pp. 875-897
Author(s):  
Guo-Zheng Li
Keyword(s):  
Machine Learning ◽  
Data Processing ◽  
Clinical Data ◽  
Machine Learning Techniques ◽  
Support Vector ◽  
Data Sets ◽  
The Novel ◽  
Real World Data ◽  
Data Set ◽  
Learning Techniques

This chapter introduces great challenges and the novel machine learning techniques employed in clinical data processing. It argues that the novel machine learning techniques including support vector machines, ensemble learning, feature selection, feature reuse by using multi-task learning, and multi-label learning provide potentially more substantive solutions for decision support and clinical data analysis. The authors demonstrate the generalization performance of the novel machine learning techniques on real world data sets including one data set of brain glioma, one data set of coronary heart disease in Chinese Medicine and some tumor data sets of microarray. More and more machine learning techniques will be developed to improve analysis precision of clinical data sets.

A Brief Survey on Text Classification Using Various Machine Learning Techniques

2018 ◽  
Vol 8 (1) ◽  
pp. 14
Author(s):  
Padmavathi .S ◽  
M. Chidambaram
Keyword(s):  
Machine Learning ◽  
Text Classification ◽  
Fixed Number ◽  
Machine Learning Techniques ◽  
Online Information ◽  
Rule Based ◽  
Learning Techniques ◽  
Machine Learning Approach ◽  
Rule Based Approach

Text classification has grown into more significant in managing and organizing the text data due to tremendous growth of online information. It does classification of documents in to fixed number of predefined categories. Rule based approach and Machine learning approach are the two ways of text classification. In rule based approach, classification of documents is done based on manually defined rules. In Machine learning based approach, classification rules or classifier are defined automatically using example documents. It has higher recall and quick process. This paper shows an investigation on text classification utilizing different machine learning techniques.

Selecting optimal SpMV realizations for GPUs via machine learning

2021 ◽  
pp. 109434202199073
Author(s):  
Ernesto Dufrechou ◽  
Pablo Ezzatti ◽  
Enrique S Quintana-Ortí
Keyword(s):  
Machine Learning ◽  
Sparse Matrix ◽  
Machine Learning Techniques ◽  
Optimal Method ◽  
Learning Techniques ◽  
General Rules ◽  
Machine Learning Approach ◽  
The Matrix ◽  
Time And Energy ◽  
Matrix Vector

More than 10 years of research related to the development of efficient GPU routines for the sparse matrix-vector product (SpMV) have led to several realizations, each with its own strengths and weaknesses. In this work, we review some of the most relevant efforts on the subject, evaluate a few prominent routines that are publicly available using more than 3000 matrices from different applications, and apply machine learning techniques to anticipate which SpMV realization will perform best for each sparse matrix on a given parallel platform. Our numerical experiments confirm the methods offer such varied behaviors depending on the matrix structure that the identification of general rules to select the optimal method for a given matrix becomes extremely difficult, though some useful strategies (heuristics) can be defined. Using a machine learning approach, we show that it is possible to obtain unexpensive classifiers that predict the best method for a given sparse matrix with over 80% accuracy, demonstrating that this approach can deliver important reductions in both execution time and energy consumption.

scholarly journals Decision Tree: A Machine Learning for Intrusion Detection

2019 ◽  
Vol 8 (6S4) ◽  
pp. 1126-1130
Keyword(s):  
Machine Learning ◽  
Intrusion Detection ◽  
Detection System ◽  
Research Work ◽  
Machine Learning Techniques ◽  
Data Sets ◽  
Legitimate User ◽  
Learning Techniques ◽  
Three Stages

The Intrusion is a major threat to unauthorized data or legal network using the legitimate user identity or any of the back doors and vulnerabilities in the network. IDS mechanisms are developed to detect the intrusions at various levels. The objective of the research work is to improve the Intrusion Detection System performance by applying machine learning techniques based on decision trees for detection and classification of attacks. The methodology adapted will process the datasets in three stages. The experimentation is conducted on KDDCUP99 data sets based on number of features. The Bayesian three modes are analyzed for different sized data sets based upon total number of attacks. The time consumed by the classifier to build the model is analyzed and the accuracy is done.

A Hybrid Machine Learning Approach for Freeway Traffic Speed Estimation

2020 ◽  
Vol 2674 (10) ◽  
pp. 68-78
Author(s):  
Zhao Zhang ◽  
Yun Yuan ◽  
Xianfeng (Terry) Yang
Keyword(s):  
Machine Learning ◽  
Flow Model ◽  
Hybrid Approach ◽  
Traffic Monitoring ◽  
Machine Learning Techniques ◽  
Learning Approach ◽  
Freeway Traffic ◽  
Learning Techniques ◽  
Machine Learning Approach ◽  
Traffic Speed Estimation

Accurate and timely estimation of freeway traffic speeds by short segments plays an important role in traffic monitoring systems. In the literature, the ability of machine learning techniques to capture the stochastic characteristics of traffic has been proved. Also, the deployment of intelligent transportation systems (ITSs) has provided enriched traffic data, which enables the adoption of a variety of machine learning methods to estimate freeway traffic speeds. However, the limitation of data quality and coverage remain a big challenge in current traffic monitoring systems. To overcome this problem, this study aims to develop a hybrid machine learning approach, by creating a new training variable based on the second-order traffic flow model, to improve the accuracy of traffic speed estimation. Grounded on a novel integrated framework, the estimation is performed using three machine learning techniques, that is, Random Forest (RF), Extreme Gradient Boosting (XGBoost), and Artificial Neural Network (ANN). All three models are trained with the integrated dataset including the traffic flow model estimates and the iPeMS and PeMS data from the Utah Department of Transportation (DOT). Further using the PeMS data as the ground truth for model evaluation, the comparisons between the hybrid approach and pure machine learning models show that the hybrid approach can effectively capture the time-varying pattern of the traffic and help improve the estimation accuracy.

Application of multi-sensor unmanned aerial system for identification of hydrothermal alteration zones

2020 ◽  
Author(s):  
Yosoon Choi ◽  
Jieun Baek ◽  
Jangwon Suh ◽  
Sung-Min Kim
Keyword(s):  
Machine Learning ◽  
Classification Accuracy ◽  
Training Data ◽  
Sensor Data ◽  
Machine Learning Techniques ◽  
Integrated Analysis ◽  
Unmanned Aerial System ◽  
Data Sets ◽  
Learning Techniques ◽  
Hydrothermal Alteration Zones

<p>In this study, we proposed a method to utilize a multi-sensor Unmanned Aerial System (UAS) for exploration of hydrothermal alteration zones. This study selected an area (10m × 20m) composed mainly of the andesite and located on the coast, with wide outcrops and well-developed structural and mineralization elements. Multi-sensor (visible, multispectral, thermal, magnetic) data were acquired in the study area using UAS, and were studied using machine learning techniques. For utilizing the machine learning techniques, we applied the stratified random method to sample 1000 training data in the hydrothermal zone and 1000 training data in the non-hydrothermal zone identified through the field survey. The 2000 training data sets created for supervised learning were first classified into 1500 for training and 500 for testing. Then, 1500 for training were classified into 1200 for training and 300 for validation. The training and validation data for machine learning were generated in five sets to enable cross-validation. Five types of machine learning techniques were applied to the training data sets: k-Nearest Neighbors (k-NN), Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and Deep Neural Network (DNN). As a result of integrated analysis of multi-sensor data using five types of machine learning techniques, RF and SVM techniques showed high classification accuracy of about 90%. Moreover, performing integrated analysis using multi-sensor data showed relatively higher classification accuracy in all five machine learning techniques than analyzing magnetic sensing data or single optical sensing data only.</p>

scholarly journals Delivering Precision Medicine to Patients with Spinal Cord Disorders; Insights into Applications of Bioinformatics and Machine Learning from Studies of Degenerative Cervical Myelopathy

2021 ◽  
Author(s):  
Kalum J. Ost ◽  
David W. Anderson ◽  
David W. Cadotte
Keyword(s):  
Machine Learning ◽  
Precision Medicine ◽  
New Technologies ◽  
Machine Learning Techniques ◽  
Massive Datasets ◽  
Learning Framework ◽  
Learning Techniques ◽  
Machine Learning Approach ◽  
Spinal Cord Disorders ◽  
Degenerative Cervical Myelopathy

With the common adoption of electronic health records and new technologies capable of producing an unprecedented scale of data, a shift must occur in how we practice medicine in order to utilize these resources. We are entering an era in which the capacity of even the most clever human doctor simply is insufficient. As such, realizing “personalized” or “precision” medicine requires new methods that can leverage the massive amounts of data now available. Machine learning techniques provide one important toolkit in this venture, as they are fundamentally designed to deal with (and, in fact, benefit from) massive datasets. The clinical applications for such machine learning systems are still in their infancy, however, and the field of medicine presents a unique set of design considerations. In this chapter, we will walk through how we selected and adjusted the “Progressive Learning framework” to account for these considerations in the case of Degenerative Cervical Myeolopathy. We additionally compare a model designed with these techniques to similar static models run in “perfect world” scenarios (free of the clinical issues address), and we use simulated clinical data acquisition scenarios to demonstrate the advantages of our machine learning approach in providing personalized diagnoses.

Sign in / Sign up

Export Citation Format

Share Document