TasKar: A Research and Education Tool for Calculation and Representation of Binary Classification Performance Instruments

Concussion injuries remain a significant public health challenge. A significant unmet clinical need remains for tools that allow related physiological impairments and longer-term health risks to be identified earlier, better quantified, and more easily monitored over time. We address this challenge by combining a head-mounted wearable inertial motion unit (IMU)-based physiological vibration acceleration (“phybrata”) sensor and several candidate machine learning (ML) models. The performance of this solution is assessed for both binary classification of concussion patients and multiclass predictions of specific concussion-related neurophysiological impairments. Results are compared with previously reported approaches to ML-based concussion diagnostics. Using phybrata data from a previously reported concussion study population, four different machine learning models (Support Vector Machine, Random Forest Classifier, Extreme Gradient Boost, and Convolutional Neural Network) are first investigated for binary classification of the test population as healthy vs. concussion (Use Case 1). Results are compared for two different data preprocessing pipelines, Time-Series Averaging (TSA) and Non-Time-Series Feature Extraction (NTS). Next, the three best-performing NTS models are compared in terms of their multiclass prediction performance for specific concussion-related impairments: vestibular, neurological, both (Use Case 2). For Use Case 1, the NTS model approach outperformed the TSA approach, with the two best algorithms achieving an F1 score of 0.94. For Use Case 2, the NTS Random Forest model achieved the best performance in the testing set, with an F1 score of 0.90, and identified a wider range of relevant phybrata signal features that contributed to impairment classification compared with manual feature inspection and statistical data analysis. The overall classification performance achieved in the present work exceeds previously reported approaches to ML-based concussion diagnostics using other data sources and ML models. This study also demonstrates the first combination of a wearable IMU-based sensor and ML model that enables both binary classification of concussion patients and multiclass predictions of specific concussion-related neurophysiological impairments.

Download Full-text

On the parameter optimization of Support Vector Machines for binary classification

Journal of Integrative Bioinformatics ◽

10.1515/jib-2012-201 ◽

2012 ◽

Vol 9 (3) ◽

pp. 33-43 ◽

Cited By ~ 30

Author(s):

Paulo Gaspar ◽

Jaime Carbonell ◽

José Luís Oliveira

Keyword(s):

Support Vector Machines ◽

Binary Classification ◽

Classification Performance ◽

Biological Data ◽

Parameters Optimization ◽

Support Vector ◽

Minimal Risk ◽

Class Separation ◽

Vector Machines ◽

Analyse Data

Summary Classifying biological data is a common task in the biomedical context. Predicting the class of new, unknown information allows researchers to gain insight and make decisions based on the available data. Also, using classification methods often implies choosing the best parameters to obtain optimal class separation, and the number of parameters might be large in biological datasets.Support Vector Machines provide a well-established and powerful classification method to analyse data and find the minimal-risk separation between different classes. Finding that separation strongly depends on the available feature set and the tuning of hyper-parameters. Techniques for feature selection and SVM parameters optimization are known to improve classification accuracy, and its literature is extensive.In this paper we review the strategies that are used to improve the classification performance of SVMs and perform our own experimentation to study the influence of features and hyper-parameters in the optimization process, using several known kernels.

Download Full-text

Automated detection of pneumonia cases using deep transfer learning with paediatric chest X-ray images

British Journal of Radiology ◽

10.1259/bjr.20201263 ◽

2021 ◽

pp. 20201263

Author(s):

Mohammad Salehi ◽

Reza Mohammadi ◽

Hamed Ghaffari ◽

Nahid Sadighi ◽

Reza Reiazi

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Transfer Learning ◽

Area Under Curve ◽

Binary Classification ◽

Classification Performance ◽

Learning Approach ◽

Insufficient Data ◽

X Ray ◽

Chest X Ray

Objective: Pneumonia is a lung infection and causes the inflammation of the small air sacs (Alveoli) in one or both lungs. Proper and faster diagnosis of pneumonia at an early stage is imperative for optimal patient care. Currently, chest X-ray is considered as the best imaging modality for diagnosing pneumonia. However, the interpretation of chest X-ray images is challenging. To this end, we aimed to use an automated convolutional neural network-based transfer-learning approach to detect pneumonia in paediatric chest radiographs. Methods: Herein, an automated convolutional neural network-based transfer-learning approach using four different pre-trained models (i.e. VGG19, DenseNet121, Xception, and ResNet50) was applied to detect pneumonia in children (1–5 years) chest X-ray images. The performance of different proposed models for testing data set was evaluated using five performances metrics, including accuracy, sensitivity/recall, Precision, area under curve, and F1 score. Results: All proposed models provide accuracy greater than 83.0% for binary classification. The pre-trained DenseNet121 model provides the highest classification performance of automated pneumonia classification with 86.8% accuracy, followed by Xception model with an accuracy of 86.0%. The sensitivity of the proposed models was greater than 91.0%. The Xception and DenseNet121 models achieve the highest classification performance with F1-score greater than 89.0%. The plotted area under curve of receiver operating characteristics of VGG19, Xception, ResNet50, and DenseNet121 models are 0.78, 0.81, 0.81, and 0.86, respectively. Conclusion: Our data showed that the proposed models achieve a high accuracy for binary classification. Transfer learning was used to accelerate training of the proposed models and resolve the problem associated with insufficient data. We hope that these proposed models can help radiologists for a quick diagnosis of pneumonia at radiology departments. Moreover, our proposed models may be useful to detect other chest-related diseases such as novel Coronavirus 2019. Advances in knowledge: Herein, we used transfer learning as a machine learning approach to accelerate training of the proposed models and resolve the problem associated with insufficient data. Our proposed models achieved accuracy greater than 83.0% for binary classification.

Download Full-text

Automatic recognition of self-acknowledged limitations in clinical research literature

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocy038 ◽

2018 ◽

Vol 25 (7) ◽

pp. 855-861 ◽

Cited By ~ 4

Author(s):

Halil Kilicoglu ◽

Graciela Rosemblat ◽

Mario Malički ◽

Gerben ter Riet

Keyword(s):

Machine Learning ◽

Clinical Research ◽

Binary Classification ◽

Classification Performance ◽

Research Literature ◽

Machine Learning Algorithms ◽

Supervised Machine Learning ◽

Support Vector ◽

Rule Based ◽

Research Transparency

Abstract Objective To automatically recognize self-acknowledged limitations in clinical research publications to support efforts in improving research transparency. Methods To develop our recognition methods, we used a set of 8431 sentences from 1197 PubMed Central articles. A subset of these sentences was manually annotated for training/testing, and inter-annotator agreement was calculated. We cast the recognition problem as a binary classification task, in which we determine whether a given sentence from a publication discusses self-acknowledged limitations or not. We experimented with three methods: a rule-based approach based on document structure, supervised machine learning, and a semi-supervised method that uses self-training to expand the training set in order to improve classification performance. The machine learning algorithms used were logistic regression (LR) and support vector machines (SVM). Results Annotators had good agreement in labeling limitation sentences (Krippendorff’s α = 0.781). Of the three methods used, the rule-based method yielded the best performance with 91.5% accuracy (95% CI [90.1-92.9]), while self-training with SVM led to a small improvement over fully supervised learning (89.9%, 95% CI [88.4-91.4] vs 89.6%, 95% CI [88.1-91.1]). Conclusions The approach presented can be incorporated into the workflows of stakeholders focusing on research transparency to improve reporting of limitations in clinical studies.

Download Full-text

Boosting the classification performance of latent fingerprint segmentation using cascade of classifiers

Intelligent Decision Technologies ◽

10.3233/idt-190105 ◽

2020 ◽

Vol 14 (3) ◽

pp. 359-371 ◽

Cited By ~ 1

Author(s):

Megha Chhabra ◽

Manoj Kumar Shukla ◽

Kiran Kumar Ravulakollu

Keyword(s):

Binary Classification ◽

Poor Quality ◽

Classification Performance ◽

Latent Fingerprints ◽

Fingerprint Segmentation ◽

Binary Classifiers ◽

Maximal Independent Sets ◽

Optimal Set ◽

Optimal Feature

Segmentation and classification of latent fingerprints is a young challenging area of research. Latent fingerprints are unintentional fingermarks. These marks are ridge patterns left at crime scenes, lifted with latent or unclear view of fingermarks, making it difficult to find the guilty party. The segmentation of lifted images of such finger impressions comes with some unique challenges in domain such as poor quality images, incomplete ridge patterns, overlapping prints etc. The classification of poorly acquired data can be improved with image pre-processing, feeding all or optimal set of features extracted to suitable classifiers etc. Our classification system proposes two main steps. First, various effective extracted features are compartmentalised into maximal independent sets with high correlation value, Second, conventional supervised technique based binary classifiers are combined into a cascade/stack of classifiers. These classifiers are fed with all or optimal feature set(s) for binary classification of fingermarks as ridge patterns from non-ridge background. The experimentation shows improvement in accuracy rate on IIIT-D database with supervised algorithms.

Download Full-text

A New Algorithm for Analysis of MiRNA Expression Profiles—SVM-RFE-FKNN

Journal of Imaging Science and Technology ◽

10.2352/j.imagingsci.technol.2021.65.3.030407 ◽

2021 ◽

Author(s):

Duan Mei ◽

Qiang Liu

Keyword(s):

Mirna Expression ◽

Nearest Neighbor ◽

Expression Profiles ◽

Binary Classification ◽

Characteristic Curve ◽

Classification Performance ◽

Recursive Feature Elimination ◽

Support Vector ◽

K Nearest Neighbor ◽

Mirna Expression Profiles

Based on MicroRNA (miRNA) expression profiles, this article proposes a new algorithm—SVM-RFE-FKNN, which combines the support vector machine-recursive feature elimination (SVM-RFE) algorithm and the fuzzy K -nearest neighbor (FKNN) algorithm, to realize binary classification of tumors. First, the SVM-RFE algorithm was used to select features from the miRNA expression profile dataset to constitute feature subsets and to determine the maximum number of support vectors. Next, this maximum number was regarded as the upper limit of the parameter K in the FKNN algorithm that was then used to classify the samples to be tested. Finally, the leave-one-out cross-validation method was adopted to assess the classification performance of the proposed algorithm. Through experiments, our proposed algorithm was compared with other twelve classification methods, and the result shows that our algorithm had better classification performance. Specifically, with only a few miRNA biomarkers, the proposed algorithm could reach an accuracy of 99.46% and an area under the receiver operating characteristic curve (AUC) of 0.9874.

Download Full-text

Mental Task Evaluation for Hybrid NIRS-EEG Brain-Computer Interfaces

Computational Intelligence and Neuroscience ◽

10.1155/2017/3524208 ◽

2017 ◽

Vol 2017 ◽

pp. 1-24 ◽

Cited By ~ 9

Author(s):

Hubert Banville ◽

Rishabh Gupta ◽

Tiago H. Falk

Keyword(s):

Near Infrared ◽

Mental Arithmetic ◽

Binary Classification ◽

Time Windows ◽

Brain Regions ◽

Classification Performance ◽

Response Patterns ◽

Word Generation ◽

Nirs Data ◽

Accuracy Increase

Based on recent electroencephalography (EEG) and near-infrared spectroscopy (NIRS) studies that showed that tasks such as motor imagery and mental arithmetic induce specific neural response patterns, we propose a hybrid brain-computer interface (hBCI) paradigm in which EEG and NIRS data are fused to improve binary classification performance. We recorded simultaneous NIRS-EEG data from nine participants performing seven mental tasks (word generation, mental rotation, subtraction, singing and navigation, and motor and face imagery). Classifiers were trained for each possible pair of tasks using (1) EEG features alone, (2) NIRS features alone, and (3) EEG and NIRS features combined, to identify the best task pairs and assess the usefulness of a multimodal approach. The NIRS-EEG approach led to an average increase in peak kappa of 0.03 when using features extracted from one-second windows (equivalent to an increase of 1.5% in classification accuracy for balanced classes). The increase was much stronger (0.20, corresponding to an 10% accuracy increase) when focusing on time windows of high NIRS performance. The EEG and NIRS analyses further unveiled relevant brain regions and important feature types. This work provides a basis for future NIRS-EEG hBCI studies aiming to improve classification performance toward more efficient and flexible BCIs.

Download Full-text

Comparison of Deep-Learning and Conventional Machine-Learning Methods for the Automatic Recognition of the Hepatocellular Carcinoma Areas from Ultrasound Images

Sensors ◽

10.3390/s20113085 ◽

2020 ◽

Vol 20 (11) ◽

pp. 3085 ◽

Cited By ~ 2

Author(s):

Raluca Brehar ◽

Delia-Alexandrina Mitrea ◽

Flaviu Vancea ◽

Tiberiu Marita ◽

Sergiu Nedevschi ◽

...

Keyword(s):

Machine Learning ◽

Hepatocellular Carcinoma ◽

Deep Learning ◽

Binary Classification ◽

Classification Performance ◽

Ultrasound Images ◽

Learning Methods ◽

Machine Learning Methods ◽

Segmentation Accuracy ◽

Conventional Machine

The emergence of deep-learning methods in different computer vision tasks has proved to offer increased detection, recognition or segmentation accuracy when large annotated image datasets are available. In the case of medical image processing and computer-aided diagnosis within ultrasound images, where the amount of available annotated data is smaller, a natural question arises: are deep-learning methods better than conventional machine-learning methods? How do the conventional machine-learning methods behave in comparison with deep-learning methods on the same dataset? Based on the study of various deep-learning architectures, a lightweight multi-resolution Convolutional Neural Network (CNN) architecture is proposed. It is suitable for differentiating, within ultrasound images, between the Hepatocellular Carcinoma (HCC), respectively the cirrhotic parenchyma (PAR) on which HCC had evolved. The proposed deep-learning model is compared with other CNN architectures that have been adapted by transfer learning for the ultrasound binary classification task, but also with conventional machine-learning (ML) solutions trained on textural features. The achieved results show that the deep-learning approach overcomes classical machine-learning solutions, by providing a higher classification performance.

Download Full-text

Exploring Symmetry of Binary Classification Performance Metrics

Symmetry ◽

10.3390/sym11010047 ◽

2019 ◽

Vol 11 (1) ◽

pp. 47 ◽

Cited By ~ 1

Author(s):

Amalia Luque ◽

Alejandro Carrasco ◽

Alejandro Martín ◽

Juan Ramón Lama

Keyword(s):

Performance Metrics ◽

Binary Classification ◽

Confusion Matrix ◽

Full Range ◽

Classification Performance ◽

Classification Problems ◽

Performance Metric ◽

Selection For ◽

Proper Performance ◽

Insight Into

Selecting the proper performance metric constitutes a key issue for most classification problems in the field of machine learning. Although the specialized literature has addressed several topics regarding these metrics, their symmetries have yet to be systematically studied. This research focuses on ten metrics based on a binary confusion matrix and their symmetric behaviour is formally defined under all types of transformations. Through simulated experiments, which cover the full range of datasets and classification results, the symmetric behaviour of these metrics is explored by exposing them to hundreds of simple or combined symmetric transformations. Cross-symmetries among the metrics and statistical symmetries are also explored. The results obtained show that, in all cases, three and only three types of symmetries arise: labelling inversion (between positive and negative classes); scoring inversion (concerning good and bad classifiers); and the combination of these two inversions. Additionally, certain metrics have been shown to be independent of the imbalance in the dataset and two cross-symmetries have been identified. The results regarding their symmetries reveal a deeper insight into the behaviour of various performance metrics and offer an indicator to properly interpret their values and a guide for their selection for certain specific applications.

Download Full-text