Polarity correspondence: A general principle for performance of speeded binary classification tasks.

Women’s cancers remain a major challenge for many health systems. Between 1991 and 2017, the death rate for all major cancers fell continuously in the United States, excluding uterine cervix and uterine corpus cancers. Together with HPV (Human Papillomavirus) testing and cytology, colposcopy has played a central role in cervical cancer screening. This medical procedure allows physicians to view the cervix at a magnification of up to 10%. This paper presents an automated colposcopy image analysis framework for the classification of precancerous and cancerous lesions of the uterine cervix. This framework is based on an ensemble of MobileNetV2 networks. Our experimental results show that this method achieves accuracies of 83.33% and 91.66% on the four-class and binary classification tasks, respectively. These results are promising for the future use of automatic classification methods based on deep learning as tools to support medical doctors.

Download Full-text

The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation

BMC Genomics ◽

10.1186/s12864-019-6413-7 ◽

2020 ◽

Vol 21 (1) ◽

Cited By ~ 149

Author(s):

Davide Chicco ◽

Giuseppe Jurman

Keyword(s):

Correlation Coefficient ◽

Matthews Correlation Coefficient ◽

Binary Classification ◽

Confusion Matrix ◽

Mathematical Properties ◽

Statistical Measures ◽

Positive Elements ◽

Classification Tasks ◽

Classification Evaluation ◽

Confusion Matrices

Abstract Background To evaluate binary classifications and their confusion matrices, scientific researchers can employ several statistical rates, accordingly to the goal of the experiment they are investigating. Despite being a crucial issue in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on imbalanced datasets. Results The Matthews correlation coefficient (MCC), instead, is a more reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset. Conclusions In this article, we show how MCC produces a more informative and truthful score in evaluating binary classifications than accuracy and F1 score, by first explaining the mathematical properties, and then the asset of MCC in six synthetic use cases and in a real genomics scenario. We believe that the Matthews correlation coefficient should be preferred to accuracy and F1 score in evaluating binary classification tasks by all scientific communities.

Download Full-text

How Does Knowledge of the AUC Constrain the Set of Possible Ground-Truth Labelings?

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33015425 ◽

2019 ◽

Vol 33 ◽

pp. 5425-5432

Author(s):

Jacob Whitehill

Keyword(s):

Machine Learning ◽

Empirical Evidence ◽

Recent Work ◽

Roc Curve ◽

Binary Classification ◽

Ground Truth ◽

Mathematical Structure ◽

Test Set ◽

Classification Tasks ◽

N Vector

Recent work on privacy-preserving machine learning has considered how datamining competitions such as Kaggle could potentially be “hacked”, either intentionally or inadvertently, by using information from an oracle that reports a classifier’s accuracy on the test set (Blum and Hardt 2015; Hardt and Ullman 2014; Zheng 2015; Whitehill 2016). For binary classification tasks in particular, one of the most common accuracy metrics is the Area Under the ROC Curve (AUC), and in this paper we explore the mathematical structure of how the AUC is computed from an n-vector of real-valued “guesses” with respect to the ground-truth labels. Under the assumption of perfect knowledge of the test set AUC c=p/q, we show how knowing c constrains the set W of possible ground-truth labelings, and we derive an algorithm both to compute the exact number of such labelings and to enumerate efficiently over them. We also provide empirical evidence that, surprisingly, the number of compatible labelings can actually decrease as n grows, until a test set-dependent threshold is reached. Finally, we show how W can be efficiently whittled down, through pairs of oracle queries, to infer all the groundtruth test labels with complete certainty.

Download Full-text

A Multi-objective Optimisation Design Approach for Ensemble Member Combination in Binary Classification Tasks

10.17648/sbai-2019-111122 ◽

2019 ◽

Author(s):

Victor Henrique Alves Ribeiro ◽

Matheus Henrique Dal Molin Ribeiro ◽

Leandro dos Santos Coelho ◽

Gilberto Reynoso Meza

Keyword(s):

Binary Classification ◽

Ensemble Member ◽

Design Approach ◽

Multi Objective ◽

Classification Tasks

Download Full-text

The Eighty Five Percent Rule for Optimal Learning

10.1101/255182 ◽

2018 ◽

Cited By ~ 2

Author(s):

Robert C. Wilson ◽

Amitai Shenhav ◽

Mark Straccia ◽

Jonathan D. Cohen

Keyword(s):

Neural Networks ◽

Gradient Descent ◽

Broad Class ◽

Binary Classification ◽

Learning Algorithms ◽

Sweet Spot ◽

Optimal Learning ◽

Rate Of Learning ◽

Classification Tasks

AbstractResearchers and educators have long wrestled with the question of how best to teach their clients be they human, animal or machine. Here we focus on the role of a single variable, the difficulty of training, and examine its effect on the rate of learning. In many situations we find that there is a sweet spot in which training is neither too easy nor too hard, and where learning progresses most quickly. We derive conditions for this sweet spot for a broad class of learning algorithms in the context of binary classification tasks, in which ambiguous stimuli must be sorted into one of two classes. For all of these gradient-descent based learning algorithms we find that the optimal error rate for training is around 15.87% or, conversely, that the optimal training accuracy is about 85%. We demonstrate the efficacy of this ‘Eighty Five Percent Rule’ for artificial neural networks used in AI and biologically plausible neural networks thought to describe human and animal learning.

Download Full-text

An SVM-Based Framework for Long-Term Learning Systems

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33019915 ◽

2019 ◽

Vol 33 ◽

pp. 9915-9916

Author(s):

Diana Benavides-Prado

Keyword(s):

Binary Classification ◽

Upper Bounds ◽

Support Vector ◽

Transfer Of Knowledge ◽

Learning Framework ◽

Vector Machines ◽

Novel Method ◽

Classification Tasks ◽

Retention Of Knowledge

In our research, we study the problem of learning a sequence of supervised tasks. This is a long-standing challenge in machine learning. Our work relies on transfer of knowledge between hypotheses learned with Support Vector Machines. Transfer occurs in two directions: forward and backward. We have proposed to selectively transfer forward support vector coefficients from previous hypotheses as upper-bounds on support vector coefficients to be learned on a target task. We also proposed a novel method for refining existing hypotheses by transferring backward knowledge from a target hypothesis learned recently. We have improved this method through a hypothesis refinement approach that refines whilst encouraging retention of knowledge. Our contribution is represented in a long-term learning framework for binary classification tasks received sequentially one at a time.

Download Full-text

The evaluation of binary classification tasks in economical prediction

Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis ◽

10.11118/actaun201058060369 ◽

2010 ◽

Vol 58 (6) ◽

pp. 369-378

Author(s):

Martin Pokorný

Keyword(s):

False Positive ◽

Roc Analysis ◽

Binary Classification ◽

False Negative ◽

Optimal Threshold ◽

False Negatives ◽

Classifier Performance ◽

False Negative Error ◽

Classification Tasks ◽

Error Costs

In the area of economical classification tasks, the accuracy maximization is often used to evaluate classifier performance. Accuracy maximization (or error rate minimization) suffers from the assumption of equal false positive and false negative error costs. Furthermore, accuracy is not able to express true classifier performance under skewed class distribution. Due to these limitations, the use of accuracy on real tasks is questionable. In a real binary classification task, the difference between the costs of false positive and false negative error is usually critical. To overcome this issue, the Receiver Operating Characteristic (ROC) method in relation to decision-analytic principles can be used. One essential advantage of this method is the possibility of classifier performance visualization by means of a ROC graph. This paper presents concrete examples of binary classification, where the inadequacy of accuracy as the evaluation metric is shown, and on the same examples the ROC method is applied. From the set of possible classification models, the probabilistic classifier with continuous output is under consideration. Mainly two questions are solved. Firstly, the selection of the best classifier from a set of possible classifiers. For example, accuracy metric rates two classifiers almost equivalently (87.7 % and 89.3 %), whereas decision analysis (via costs minimization) or ROC analysis reveal different performance according to target conditions of unequal error costs of false positives and false negatives. Secondly, the setting of an optimal decision threshold at classifier’s output. For example, accuracy maximization finds the optimal threshold at classifier’s output in value of 0.597, but the optimal threshold respecting higher costs of false negatives is discovered by costs minimization or ROC analysis in a value substantially lower (0.477).

Download Full-text

3D CNN with Visual Insights for Early Detection of Lung Cancer Using Gradient-Weighted Class Activation

Journal of Healthcare Engineering ◽

10.1155/2021/6695518 ◽

2021 ◽

Vol 2021 ◽

pp. 1-11

Author(s):

Eali Stephen Neal Joshua ◽

Debnath Bhattacharyya ◽

Midhun Chakkravarthy ◽

Yung-Cheol Byun

Keyword(s):

Low Dose ◽

Binary Classification ◽

Lung Nodule ◽

Fine Grained ◽

Internal Working ◽

Nodule Detection ◽

Classification Tasks ◽

3D Cnn ◽

Nodule Classification ◽

Fold Cross Validation

The 3D convolutional neural network is able to make use of the full nonlinear 3D context information of lung nodule detection from the DICOM (Digital Imaging and Communications in Medicine) images, and the Gradient Class Activation has shown to be useful for tailoring classification tasks and localization interpretation for fine-grained features and visual explanation for the internal working. Gradient-weighted class activation plays a crucial role for clinicians and radiologists in terms of trusting and adopting the model. Practitioners not only rely on a model that can provide high precision but also really want to gain the respect of radiologists. So, in this paper, we explored the lung nodule classification using the improvised 3D AlexNet with lightweight architecture. Our network employed the full nature of the multiview network strategy. We have conducted the binary classification (benign and malignant) on computed tomography (CT) images from the LUNA 16 database conglomerate and database image resource initiative. The results obtained are through the 10-fold cross-validation. Experimental results have shown that the proposed lightweight architecture achieved a superior classification accuracy of 97.17% on LUNA 16 dataset when compared with existing classification algorithms and low-dose CT scan images as well.

Download Full-text