scholarly journals The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation

BMC Genomics ◽  
2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Davide Chicco ◽  
Giuseppe Jurman
Keyword(s):  
Correlation Coefficient ◽  
Matthews Correlation Coefficient ◽  
Binary Classification ◽  
Confusion Matrix ◽  
Mathematical Properties ◽  
Statistical Measures ◽  
Positive Elements ◽  
Classification Tasks ◽  
Classification Evaluation ◽  
Confusion Matrices

Abstract Background To evaluate binary classifications and their confusion matrices, scientific researchers can employ several statistical rates, accordingly to the goal of the experiment they are investigating. Despite being a crucial issue in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on imbalanced datasets. Results The Matthews correlation coefficient (MCC), instead, is a more reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset. Conclusions In this article, we show how MCC produces a more informative and truthful score in evaluating binary classifications than accuracy and F1 score, by first explaining the mathematical properties, and then the asset of MCC in six synthetic use cases and in a real genomics scenario. We believe that the Matthews correlation coefficient should be preferred to accuracy and F1 score in evaluating binary classification tasks by all scientific communities.

scholarly journals The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

2021 ◽  
Vol 14 (1) ◽  
Author(s):  
Davide Chicco ◽  
Niklas Tötsch ◽  
Giuseppe Jurman
Keyword(s):  
Correlation Coefficient ◽  
Matthews Correlation Coefficient ◽  
Confusion Matrix ◽  
General Purpose ◽  
Critical Conditions ◽  
Statistical Indicator ◽  
Equal Importance ◽  
Classifier Performance ◽  
Mathematical Relationships ◽  
Confusion Matrices

AbstractEvaluating binary classifications is a pivotal task in statistics and machine learning, because it can influence decisions in multiple areas, including for example prognosis or therapies of patients in critical conditions. The scientific community has not agreed on a general-purpose statistical indicator for evaluating two-class confusion matrices (having true positives, true negatives, false positives, and false negatives) yet, even if advantages of the Matthews correlation coefficient (MCC) over accuracy and F1 score have already been shown.In this manuscript, we reaffirm that MCC is a robust metric that summarizes the classifier performance in a single value, if positive and negative cases are of equal importance. We compare MCC to other metrics which value positive and negative cases equally: balanced accuracy (BA), bookmaker informedness (BM), and markedness (MK). We explain the mathematical relationships between MCC and these indicators, then show some use cases and a bioinformatics scenario where these metrics disagree and where MCC generates a more informative response.Additionally, we describe three exceptions where BM can be more appropriate: analyzing classifications where dataset prevalence is unrepresentative, comparing classifiers on different datasets, and assessing the random guessing level of a classifier. Except in these cases, we believe that MCC is the most informative among the single metrics discussed, and suggest it as standard measure for scientists of all fields. A Matthews correlation coefficient close to +1, in fact, means having high values for all the other confusion matrix metrics. The same cannot be said for balanced accuracy, markedness, bookmaker informedness, accuracy and F1 score.

scholarly journals Comparative study of quality estimation of binary classification

Informatics ◽  
2020 ◽  
Vol 17 (1) ◽  
pp. 87-101
Author(s):  
V. V. Starovoitov ◽  
Yu. I. Golub
Keyword(s):  
Matthews Correlation Coefficient ◽  
Binary Classification ◽  
Confusion Matrix ◽  
Diagnostic Odds Ratio ◽  
Quality Estimation ◽  
Imbalanced Datasets ◽  
Error Matrix ◽  
Classification Errors ◽  
Sensitivity Specificity

The paper describes results of analytical and experimental analysis of seventeen functions used for evaluation of binary classification results of arbitrary data. The results are presented by 2×2 error matrices. The behavior and properties of the main functions calculated by the elements of such matrices are studied.  Classification options with balanced and imbalanced datasets are analyzed. It is shown that there are linear dependencies between some functions, many functions are invariant to the transposition of the error matrix, which allows us to calculate the estimation without specifying the order in which their elements were written to the matrices.It has been proven that all classical measures such as Sensitivity, Specificity, Precision, Accuracy, F1, F2, GM, the Jacquard index are sensitive to the imbalance of classified data and distort estimation of smaller class objects classification errors. Sensitivity to imbalance is found in the Matthews correlation coefficient and Kohen’s kappa. It has been experimentally shown that functions such as the confusion entropy, the discriminatory power, and the diagnostic odds ratio should not be used for analysis of binary classification of imbalanced datasets. The last two functions are invariant to the imbalance of classified data, but poorly evaluate results with approximately equal common percentage of classification errors in two classes.We proved that the area under the ROC curve (AUC) and the Yuden index calculated from the binary classification confusion matrix are linearly dependent and are the best estimation functions of both balanced and imbalanced datasets.

Bases for Strategy Choice and Strategy Transitions in Binary Classification Tasks

2004 ◽  
Author(s):  
Lyle E. Bourne ◽  
Alice F. Healy ◽  
James A. Kole ◽  
William D. Raymond
Keyword(s):  
Binary Classification ◽  
Strategy Choice ◽  
Classification Tasks

Optimization of K Value in KNN Algorithm for Spam and Ham Email Classification

2020 ◽  
Vol 4 (2) ◽  
pp. 377-383
Author(s):  
Eko Laksono ◽  
Achmad Basuki ◽  
Fitra Bachtiar
Keyword(s):  
Frequency Distribution ◽  
Confusion Matrix ◽  
High Accuracy ◽  
K Value ◽  
A Value ◽  
Optimal Value ◽  
Classification Evaluation ◽  
Email Spam ◽  
Email Classification

There are many cases of email abuse that have the potential to harm others. This email abuse is commonly known as spam, which contains advertisements, phishing scams, and even malware. This study purpose to know the classification of email spam with ham using the KNN method as an effort to reduce the amount of spam. KNN can classify spam or ham in an email by checking it using a different K value approach. The results of the classification evaluation using confusion matrix resulted in the KNN method with a value of K = 1 having the highest accuracy value of 91.4%. From the results of the study, it is known that the optimization of the K value in KNN using frequency distribution clustering can produce high accuracy of 100%, while k-means clustering produces an accuracy of 99%. So based on the results of the existing accuracy values, the frequency distribution clustering and k-means clustering can be used to optimize the K-optimal value of the KNN in the classification of existing spam emails.

Improved Correlation Coefficients of Quadripartitioned Single-Valued Neutrosophic Sets and Interval-Quadripartitioned Neutrosophic Sets

2020 ◽  
pp. 331-363
Author(s):  
Mohanasundari M. ◽  
Mohana K.
Keyword(s):  
Decision Making ◽  
Correlation Coefficient ◽  
Correlation Coefficients ◽  
Multiple Attribute Decision Making ◽  
Neutrosophic Sets ◽  
Statistical Measures ◽  
Multiple Attribute

A correlation coefficient is one of the statistical measures that helps to find the degree of changes to the value of one variable predict change to the value of another. Quadripartitioned single valued neutrosophic sets is an improvization of Wang's single valued neutrosophic sets. This chapter deals the improved correlation coefficients of quadripartitioned single valued neutrosophic sets, interval quadripartitioned neutrosophic sets, and investigates its properties. And this concept is also applied in multiple-attribute decision-making methods with quadripartitioned single valued neutrosophic environment and interval quadripartitioned neutrosophic environment. Finally an illustrated example is given in the proposed method to the multiple-attribute decision-making problems.

Sign in / Sign up

Export Citation Format

Share Document