scholarly journals Better Classifier Calibration for Small Datasets

2020 ◽  
Vol 14 (3) ◽  
pp. 1-19 ◽  
Author(s):  
Alasalmi Tuomo ◽  
Jaakko Suutala ◽  
Juha Röning ◽  
Heli Koskimäki
Keyword(s):  
Classifier Calibration

Classifier Calibration

2017 ◽  
pp. 210-217
Author(s):  
Peter A. Flach
Keyword(s):  
Classifier Calibration

scholarly journals Calibration of Machine Learning Models

2010 ◽  
pp. 128-146 ◽  
Author(s):  
Antonio Bella ◽  
Cèsar Ferri ◽  
José Hernández-Orallo ◽  
María José Ramírez-Quintana
Keyword(s):  
Machine Learning ◽  
Single Case ◽  
Machine Learning Techniques ◽  
Average Error ◽  
Learning Models ◽  
Classifier Calibration ◽  
Learning Techniques ◽  
Probabilistic Classifier ◽  
Calibration Techniques ◽  
Machine Learning Models

The evaluation of machine learning models is a crucial step before their application because it is essential to assess how well a model will behave for every single case. In many real applications, not only is it important to know the “total” or the “average” error of the model, it is also important to know how this error is distributed and how well confidence or probability estimations are made. Many current machine learning techniques are good in overall results but have a bad distribution assessment of the error. For these cases, calibration techniques have been developed as postprocessing techniques in order to improve the probability estimation or the error distribution of an existing model. This chapter presents the most common calibration techniques and calibration measures. Both classification and regression are covered, and a taxonomy of calibration techniques is established. Special attention is given to probabilistic classifier calibration.

Evidential Logistic Regression for Binary SVM Classifier Calibration

2014 ◽  
pp. 49-57 ◽  
Author(s):  
Philippe Xu ◽  
Franck Davoine ◽  
Thierry Denoeux
Keyword(s):  
Logistic Regression ◽  
Svm Classifier ◽  
Classifier Calibration

scholarly journals Automatic Fungi Recognition: Deep Learning Meets Mycology

Sensors ◽  
2022 ◽  
Vol 22 (2) ◽  
pp. 633
Author(s):  
Lukáš Picek ◽  
Milan Šulc ◽  
Jiří Matas ◽  
Jacob Heilmann-Clausen ◽  
Thomas S. Jeppesen ◽  
...  
Keyword(s):  
Current System ◽  
Species Recognition ◽  
Recognition System ◽  
Classification Performance ◽  
Fine Grained ◽  
Human In The Loop ◽  
Classifier Calibration ◽  
Virtuous Cycle ◽  
Real Time Identification ◽  
The Impact

The article presents an AI-based fungi species recognition system for a citizen-science community. The system’s real-time identification too — FungiVision — with a mobile application front-end, led to increased public interest in fungi, quadrupling the number of citizens collecting data. FungiVision, deployed with a human-in-the-loop, reaches nearly 93% accuracy. Using the collected data, we developed a novel fine-grained classification dataset — Danish Fungi 2020 (DF20) — with several unique characteristics: species-level labels, a small number of errors, and rich observation metadata. The dataset enables the testing of the ability to improve classification using metadata, e.g., time, location, habitat and substrate, facilitates classifier calibration testing and finally allows the study of the impact of the device settings on the classification performance. The continual flow of labelled data supports improvements of the online recognition system. Finally, we present a novel method for the fungi recognition service, based on a Vision Transformer architecture. Trained on DF20 and exploiting available metadata, it achieves a recognition error that is 46.75% lower than the current system. By providing a stream of labeled data in one direction, and an accuracy increase in the other, the collaboration creates a virtuous cycle helping both communities.

Incorporation of Inter-Subject Information to Improve the Accuracy of Subject-Specific P300 Classifiers

2016 ◽  
Vol 26 (03) ◽  
pp. 1650010 ◽  
Author(s):  
Minpeng Xu ◽  
Jing Liu ◽  
Long Chen ◽  
Hongzhi Qi ◽  
Feng He ◽  
...  
Keyword(s):  
Brain Computer Interface ◽  
Heterogeneous Data ◽  
Computer Interface ◽  
Training Dataset ◽  
Comprehensive Investigation ◽  
P300 Speller ◽  
Classifier Calibration ◽  
Accuracy And Precision ◽  
Subject Specific ◽  
The Subject

Although the inter-subject information has been demonstrated to be effective for a rapid calibration of the P300-based brain–computer interface (BCI), it has never been comprehensively tested to find if the incorporation of heterogeneous data could enhance the accuracy. This study aims to improve the subject-specific P300 classifier by adding other subject’s data. A classifier calibration strategy, weighted ensemble learning generic information (WELGI), was developed, in which elementary classifiers were constructed by using both the intra- and inter-subject information and then integrated into a strong classifier with a weight assessment. 55 subjects were recruited to spell 20 characters offline using the conventional P300-based BCI, i.e. the P300-speller. Four different metrics, the P300 accuracy and precision, the round accuracy, and the character accuracy, were performed for a comprehensive investigation. The results revealed that the classifier constructed on the training dataset in combination with adding other subject’s data was significantly superior to that without the inter-subject information. Therefore, the WELGI is an effective classifier calibration strategy which uses the inter-subject information to improve the accuracy of subject-specific P300 classifiers, and could also be applied to other BCI paradigms.

Classifier calibration using splined empirical probabilities in clinical risk prediction

2014 ◽  
Vol 18 (2) ◽  
pp. 156-165 ◽  
Author(s):  
René Gaudoin ◽  
Giovanni Montana ◽  
Simon Jones ◽  
Paul Aylin ◽  
Alex Bottle
Keyword(s):  
Risk Prediction ◽  
Clinical Risk ◽  
Classifier Calibration

scholarly journals Calibration of Machine Learning Models

2012 ◽  
pp. 32-49
Author(s):  
Antonio Bella ◽  
Cèsar Ferri ◽  
José Hernández-Orallo ◽  
María José Ramírez-Quintana
Keyword(s):  
Machine Learning ◽  
Single Case ◽  
Machine Learning Techniques ◽  
Average Error ◽  
Learning Models ◽  
Classifier Calibration ◽  
Learning Techniques ◽  
Probabilistic Classifier ◽  
Calibration Techniques ◽  
Machine Learning Models

The evaluation of machine learning models is a crucial step before their application because it is essential to assess how well a model will behave for every single case. In many real applications, not only is it important to know the “total” or the “average” error of the model, it is also important to know how this error is distributed and how well confidence or probability estimations are made. Many current machine learning techniques are good in overall results but have a bad distribution assessment of the error. For these cases, calibration techniques have been developed as postprocessing techniques in order to improve the probability estimation or the error distribution of an existing model. This chapter presents the most common calibration techniques and calibration measures. Both classification and regression are covered, and a taxonomy of calibration techniques is established. Special attention is given to probabilistic classifier calibration.

Classifier Calibration

2016 ◽  
pp. 1-8 ◽  
Author(s):  
Peter A. Flach
Keyword(s):  
Classifier Calibration
Sign in / Sign up

Export Citation Format

Share Document