scholarly journals Greed Is Good: Rapid Hyperparameter Optimization and Model Selection Using Greedy k-Fold Cross Validation

Electronics ◽  
2021 ◽  
Vol 10 (16) ◽  
pp. 1973
Author(s):  
Daniel S. Soper
Keyword(s):  
Machine Learning ◽  
Model Selection ◽  
Cross Validation ◽  
Search Time ◽  
Early Stopping ◽  
Hyperparameter Optimization ◽  
Validation Method ◽  
Computational Budget ◽  
Real World Datasets ◽  
Fold Cross Validation

Selecting a final machine learning (ML) model typically occurs after a process of hyperparameter optimization in which many candidate models with varying structural properties and algorithmic settings are evaluated and compared. Evaluating each candidate model commonly relies on k-fold cross validation, wherein the data are randomly subdivided into k folds, with each fold being iteratively used as a validation set for a model that has been trained using the remaining folds. While many research studies have sought to accelerate ML model selection by applying metaheuristic and other search methods to the hyperparameter space, no consideration has been given to the k-fold cross validation process itself as a means of rapidly identifying the best-performing model. The current study rectifies this oversight by introducing a greedy k-fold cross validation method and demonstrating that greedy k-fold cross validation can vastly reduce the average time required to identify the best-performing model when given a fixed computational budget and a set of candidate models. This improved search time is shown to hold across a variety of ML algorithms and real-world datasets. For scenarios without a computational budget, this paper also introduces an early stopping algorithm based on the greedy cross validation method. The greedy early stopping method is shown to outperform a competing, state-of-the-art early stopping method both in terms of search time and the quality of the ML models selected by the algorithm. Since hyperparameter optimization is among the most time-consuming, computationally intensive, and monetarily expensive tasks in the broader process of developing ML-based solutions, the ability to rapidly identify optimal machine learning models using greedy cross validation has obvious and substantial benefits to organizations and researchers alike.

A STUDY OF EARLY STOPPING AND MODEL SELECTION APPLIED TO THE PAPERMAKING INDUSTRY

2000 ◽  
Vol 10 (01) ◽  
pp. 9-18 ◽  
Author(s):  
PETER J. EDWARDS ◽  
ALAN F. MURRAY
Keyword(s):  
Neural Network ◽  
Model Selection ◽  
Cross Validation ◽  
Model Development ◽  
Network Models ◽  
Comparison Study ◽  
Early Stopping ◽  
Neural Network Models ◽  
Model Evidence ◽  
Fold Cross Validation

This paper addresses the issues of neural network model development and maintenance in the context of a complex task taken from the papermaking industry. In particular, it describes a comparison study of early stopping techniques and model selection, both to optimise neural network models for generalisation performance. The results presented here show that early stopping via use of a Bayesian model evidence measure is a viable way of optimising performance while also making maximum use of all the data. In addition, they show that ten-fold cross-validation performs well as a model selector and as an estimator of prediction accuracy. These results are important in that they show how neural network models may be optimally trained and selected for highly complex industrial tasks where the data are noisy and limited in number.

Prediction of K562 Cells Functional Inhibitors Based on Machine Learning Approaches

2020 ◽  
Vol 25 (40) ◽  
pp. 4296-4302 ◽  
Author(s):  
Yuan Zhang ◽  
Zhenyan Han ◽  
Qian Gao ◽  
Xiaoyi Bai ◽  
Chi Zhang ◽  
...  
Keyword(s):  
Machine Learning ◽  
Inclusion Bodies ◽  
Cross Validation ◽  
Independent Set ◽  
K562 Cells ◽  
Machine Learning Algorithms ◽  
Learning Approaches ◽  
Validation Test ◽  
Excess Number ◽  
Fold Cross Validation

Background: β thalassemia is a common monogenic genetic disease that is very harmful to human health. The disease arises is due to the deletion of or defects in β-globin, which reduces synthesis of the β-globin chain, resulting in a relatively excess number of α-chains. The formation of inclusion bodies deposited on the cell membrane causes a decrease in the ability of red blood cells to deform and a group of hereditary haemolytic diseases caused by massive destruction in the spleen. Methods: In this work, machine learning algorithms were employed to build a prediction model for inhibitors against K562 based on 117 inhibitors and 190 non-inhibitors. Results: The overall accuracy (ACC) of a 10-fold cross-validation test and an independent set test using Adaboost were 83.1% and 78.0%, respectively, surpassing Bayes Net, Random Forest, Random Tree, C4.5, SVM, KNN and Bagging. Conclusion: This study indicated that Adaboost could be applied to build a learning model in the prediction of inhibitors against K526 cells.

scholarly journals Evaluation and Identification of the Neuroprotective Compounds of Xiaoxuming Decoction by Machine Learning: A Novel Mode to Explore the Combination Rules in Traditional Chinese Medicine Prescription

2019 ◽  
Vol 2019 ◽  
pp. 1-14
Author(s):  
Shilun Yang ◽  
Yanjia Shen ◽  
Wendan Lu ◽  
Yinglin Yang ◽  
Haigang Wang ◽  
...  
Keyword(s):  
Machine Learning ◽  
Chinese Medicine ◽  
Traditional Chinese Medicine ◽  
Cross Validation ◽  
Bayesian Models ◽  
Machine Learning Algorithms ◽  
Therapeutic Effects ◽  
Test Set ◽  
Screening Experiments ◽  
Fold Cross Validation

Xiaoxuming decoction (XXMD), a classic traditional Chinese medicine (TCM) prescription, has been used as a therapeutic in the treatment of stroke in clinical practice for over 1200 years. However, the pharmacological mechanisms of XXMD have not yet been elucidated. The purpose of this study was to develop neuroprotective models for identifying neuroprotective compounds in XXMD against hypoxia-induced and H2O2-induced brain cell damage. In this study, a phenotype-based classification method was designed by machine learning to identify neuroprotective compounds and to clarify the compatibility of XXMD components. Four different single classifiers (AB, kNN, CT, and RF) and molecular fingerprint descriptors were used to construct stacked naïve Bayesian models. Among them, the RF algorithm had a better performance with an average MCC value of 0.725±0.014 and 0.774±0.042 from 5-fold cross-validation and test set, respectively. The probability values calculated by four models were then integrated into a stacked Bayesian model. In total, two optimal models, s-NB-1-LPFP6 and s-NB-2-LPFP6, were obtained. The two validated optimal models revealed Matthews correlation coefficients (MCC) of 0.968 and 0.993 for 5-fold cross-validation and of 0.874 and 0.959 for the test set, respectively. Furthermore, the two models were used for virtual screening experiments to identify neuroprotective compounds in XXMD. Ten representative compounds with potential therapeutic effects against the two phenotypes were selected for further cell-based assays. Among the selected compounds, two compounds significantly inhibited H2O2-induced and Na2S2O4-induced neurotoxicity simultaneously. Together, our findings suggested that machine learning algorithms such as combination Bayesian models were feasible to predict neuroprotective compounds and to preliminarily demonstrate the pharmacological mechanisms of TCM.

scholarly journals North American Hardwoods Identification Using Machine-Learning

Forests ◽  
2020 ◽  
Vol 11 (3) ◽  
pp. 298 ◽  
Author(s):  
Dercilio Junior Verly Lopes ◽  
Greg W. Burgreen ◽  
Edward D. Entsminger
Keyword(s):  
Machine Learning ◽  
North American ◽  
Mobile Application ◽  
Cross Validation ◽  
Data Augmentation ◽  
Technical Note ◽  
Machine Learning Method ◽  
Training Set ◽  
Hardwood Species ◽  
Fold Cross Validation

This technical note determines the feasibility of using an InceptionV4_ResNetV2 convolutional neural network (CNN) to correctly identify hardwood species from macroscopic images. The method is composed of a commodity smartphone fitted with a 14× macro lens for photography. The end-grains of ten different North American hardwood species were photographed to create a dataset of 1869 images. The stratified 5-fold cross-validation machine-learning method was used, in which the number of testing samples varied from 341 to 342. Data augmentation was performed on-the-fly for each training set by rotating, zooming, and flipping images. It was found that the CNN could correctly identify hardwood species based on macroscopic images of its end-grain with an adjusted accuracy of 92.60%. With the current growing of machine-learning field, this model can then be readily deployed in a mobile application for field wood identification.

scholarly journals Classification of Brain Tumors from MRI Images Using a Convolutional Neural Network

2020 ◽  
Vol 10 (6) ◽  
pp. 1999 ◽  
Author(s):  
Milica M. Badža ◽  
Marko Č. Barjaktarović
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Brain Tumors ◽  
Convolutional Neural Network ◽  
Cross Validation ◽  
Magnetic Resonance Images ◽  
Generalization Capability ◽  
Data Set ◽  
Fold Cross Validation

The classification of brain tumors is performed by biopsy, which is not usually conducted before definitive brain surgery. The improvement of technology and machine learning can help radiologists in tumor diagnostics without invasive measures. A machine-learning algorithm that has achieved substantial results in image segmentation and classification is the convolutional neural network (CNN). We present a new CNN architecture for brain tumor classification of three tumor types. The developed network is simpler than already-existing pre-trained networks, and it was tested on T1-weighted contrast-enhanced magnetic resonance images. The performance of the network was evaluated using four approaches: combinations of two 10-fold cross-validation methods and two databases. The generalization capability of the network was tested with one of the 10-fold methods, subject-wise cross-validation, and the improvement was tested by using an augmented image database. The best result for the 10-fold cross-validation method was obtained for the record-wise cross-validation for the augmented data set, and, in that case, the accuracy was 96.56%. With good generalization capability and good execution speed, the new developed CNN architecture could be used as an effective decision-support tool for radiologists in medical diagnostics.

Model selection for direct marketing: performance criteria and validation methods

2008 ◽  
Vol 26 (3) ◽  
pp. 275-292 ◽  
Author(s):  
Geng Cui ◽  
Man Leung Wong ◽  
Guichang Zhang ◽  
Lin Li
Keyword(s):  
Neural Networks ◽  
Model Selection ◽  
Cross Validation ◽  
Direct Marketing ◽  
Latent Class Models ◽  
Performance Criteria ◽  
Content Type ◽  
Validation Methods ◽  
Practical Implications ◽  
Fold Cross Validation

PurposeThe purpose of this paper is to assess the performance of competing methods and model selection, which are non‐trivial issues given the financial implications. Researchers have adopted various methods including statistical models and machine learning methods such as neural networks to assist decision making in direct marketing. However, due to the different performance criteria and validation techniques currently in practice, comparing different methods is often not straightforward.Design/methodology/approachThis study compares the performance of neural networks with that of classification and regression tree, latent class models and logistic regression using three criteria – simple error rate, area under the receiver operating characteristic curve (AUROC), and cumulative lift – and two validation methods, i.e. bootstrap and stratified k‐fold cross‐validation. Systematic experiments are conducted to compare their performance.FindingsThe results suggest that these methods vary in performance across different criteria and validation methods. Overall, neural networks outperform the others in AUROC value and cumulative lifts, and the stratified ten‐fold cross‐validation produces more accurate results than bootstrap validation.Practical implicationsTo select predictive models to support direct marketing decisions, researchers need to adopt appropriate performance criteria and validation procedures.Originality/valueThe study addresses the key issues in model selection, i.e. performance criteria and validation methods, and conducts systematic analyses to generate the findings and practical implications.

scholarly journals Issues in performance evaluation for host–pathogen protein interaction prediction

2016 ◽  
Vol 14 (03) ◽  
pp. 1650011 ◽  
Author(s):  
Wajid Arshad Abbasi ◽  
Fayyaz Ul Amir Afsar Minhas
Keyword(s):  
Machine Learning ◽  
Protein Interactions ◽  
Cross Validation ◽  
Protein Protein Interactions ◽  
Evaluation Scheme ◽  
Host Pathogen ◽  
Pathogen Protein ◽  
Protein Interaction Prediction ◽  
Underlying Mechanisms ◽  
Fold Cross Validation

The study of interactions between host and pathogen proteins is important for understanding the underlying mechanisms of infectious diseases and for developing novel therapeutic solutions. Wet-lab techniques for detecting protein–protein interactions (PPIs) can benefit from computational predictions. Machine learning is one of the computational approaches that can assist biologists by predicting promising PPIs. A number of machine learning based methods for predicting host–pathogen interactions (HPI) have been proposed in the literature. The techniques used for assessing the accuracy of such predictors are of critical importance in this domain. In this paper, we question the effectiveness of K-fold cross-validation for estimating the generalization ability of HPI prediction for proteins with no known interactions. K-fold cross-validation does not model this scenario, and we demonstrate a sizable difference between its performance and the performance of an alternative evaluation scheme called leave one pathogen protein out (LOPO) cross-validation. LOPO is more effective in modeling the real world use of HPI predictors, specifically for cases in which no information about the interacting partners of a pathogen protein is available during training. We also point out that currently used metrics such as areas under the precision-recall or receiver operating characteristic curves are not intuitive to biologists and propose simpler and more directly interpretable metrics for this purpose.

scholarly journals Analisis Sentimen Pada Maskapai Penerbangan di Platform Twitter Menggunakan Algoritma Support Vector Machine (SVM)

Teknika ◽  
2021 ◽  
Vol 10 (1) ◽  
pp. 18-26
Author(s):  
Hendry Cipta Husada ◽  
Adi Suryaputra Paramita
Keyword(s):  
Machine Learning ◽  
Social Media ◽  
Support Vector Machine ◽  
Cross Validation ◽  
Support Vector ◽  
Learning Approach ◽  
Social Media Platform ◽  
Machine Learning Approach ◽  
Media Platform ◽  
Fold Cross Validation

Perkembangan teknologi saat ini telah memberikan kemudahan bagi banyak orang dalam mendapatkan dan menyebarkan informasi di berbagai social media platform. Twitter merupakan salah satu media yang kerap digunakan untuk menyampaikan opini sebagai bentuk reaksi seseorang atas suatu hal. Opini yang terdapat di Twitter dapat digunakan perusahaan maskapai penerbangan sebagai parameter kunci untuk mengetahui tingkat kepuasan publik sekaligus bahan evaluasi bagi perusahaan. Berdasarkan hal tersebut, diperlukan sebuah metode yang dapat secara otomatis melakukan klasifikasi opini ke dalam kategori positif, negatif, atau netral melalui proses analisis sentimen. Proses analisis sentimen dilakukan dengan proses data preprocessing, pembobotan kata menggunakan metode TF-IDF, penerapan algoritma, dan pembahasan atas hasil klasifikasi. Klasifikasi opini dilakukan dengan machine learning approach memanfaatkan algoritma multi-class Support Vector Machine (SVM). Data yang digunakan dalam penelitian ini adalah opini dalam bahasa Inggris dari para pengguna Twitter terhadap maskapai penerbangan. Berdasarkan pengujian yang telah dilakukan, hasil klasifikasi terbaik diperoleh menggunakan SVM kernel RBF pada nilai parameter 𝐶(complexity) = 10 dan 𝛾(gamma) = 1, dengan nilai accuracy sebesar 84,37% dan 80,41% ketika menggunakan 10-fold cross validation.

Sign in / Sign up

Export Citation Format

Share Document