Notes on the H-measure of classifier performance

The H-measure is a classifier performance measure which takes into account the context of application without requiring a rigid value of relative misclassification costs to be set. Since its introduction in 2009 it has become widely adopted. This paper answers various queries which users have raised since its introduction, including questions about its interpretation, the choice of a weighting function, whether it is strictly proper, its coherence, and relates the measure to other work.

Inflated 3D ConvNet context analysis for violence detection

According to the Wall Street Journal, one billion surveillance cameras will be deployed around the world by 2021. This amount of information can be hardly managed by humans. Using a Inflated 3D ConvNet as backbone, this paper introduces a novel automatic violence detection approach that outperforms state-of-the-art existing proposals. Most of those proposals consider a pre-processing step to only focus on some regions of interest in the scene, i.e., those actually containing a human subject. In this regard, this paper also reports the results of an extensive analysis on whether and how the context can affect or not the adopted classifier performance. The experiments show that context-free footage yields substantial deterioration of the classifier performance (2% to 5%) on publicly available datasets. However, they also demonstrate that performance stabilizes in context-free settings, no matter the level of context restriction applied. Finally, a cross-dataset experiment investigates the generalizability of results obtained in a single-collection experiment (same dataset used for training and testing) to cross-collection settings (different datasets used for training and testing).

Convolutional Neural Networks for Breast Density Classification: Performance and Explanation Insights

We propose and evaluate a procedure for the explainability of a breast density deep learning based classifier. A total of 1662 mammography exams labeled according to the BI-RADS categories of breast density was used. We built a residual Convolutional Neural Network, trained it and studied the responses of the model to input changes, such as different distributions of class labels in training and test sets and suitable image pre-processing. The aim was to identify the steps of the analysis with a relevant impact on the classifier performance and on the model explainability. We used the grad-CAM algorithm for CNN to produce saliency maps and computed the Spearman’s rank correlation between input images and saliency maps as a measure of explanation accuracy. We found that pre-processing is critical not only for accuracy, precision and recall of a model but also to have a reasonable explanation of the model itself. Our CNN reaches good performances compared to the state-of-art and it considers the dense pattern to make the classification. Saliency maps strongly correlate with the dense pattern. This work is a starting point towards the implementation of a standard framework to evaluate both CNN performances and the explainability of their predictions in medical image classification problems.

A Model Visualization-based Approach for Insight into Waveforms and Spectra Learned by CNNs

Recent years have shown a growth in the application of deep learning architectures such as convolutional neural networks (CNNs), to electrophysiology analysis. However, using neural networks with raw time-series data makes explainability a significant challenge. Multiple explainability approaches have been developed for insight into the spectral features learned by CNNs from EEG. However, across electrophysiology modalities, and even within EEG, there are many unique waveforms of clinical relevance. Existing methods that provide insight into waveforms learned by CNNs are of questionable utility. In this study, we present a novel model visualization-based approach that analyzes the filters in the first convolutional layer of the network. To our knowledge, this is the first method focused on extracting explainable information from EEG waveforms learned by CNNs while also providing insight into the learned spectral features. We demonstrate the viability of our approach within the context of automated sleep stage classification, a well-characterized domain that can help validate our approach. We identify 3 subgroups of filters with distinct spectral properties, determine the relative importance of each group of filters, and identify several unique waveforms learned by the classifier that were vital to the classifier performance. Our approach represents a significant step forward in explainability for electrophysiology classifiers, which we also hope will be useful for providing insights in future studies.

Clustering as feature selection method in spam classification: uncovering sick-leave sellers

PurposeThis paper aims to propose a novel way of using textual clustering as a feature selection method. It is applied to identify the most important keywords in the profile classification. The method is demonstrated through the problem of sick-leave promoters on Twitter.Design/methodology/approachFour machine learning classifiers were used on a total of 35,578 tweets posted on Twitter. The data were manually labeled into two categories: promoter and nonpromoter. Classification performance was compared when the proposed clustering feature selection approach and the standard feature selection were applied.FindingsRadom forest achieved the highest accuracy of 95.91% higher than similar work compared. Furthermore, using clustering as a feature selection method improved the Sensitivity of the model from 73.83% to 98.79%. Sensitivity (recall) is the most important measure of classifier performance when detecting promoters’ accounts that have spam-like behavior.Research limitations/implicationsThe method applied is novel, more testing is needed in other datasets before generalizing its results.Practical implicationsThe model applied can be used by Saudi authorities to report on the accounts that sell sick-leaves online.Originality/valueThe research is proposing a new way textual clustering can be used in feature selection.

Improving Performance of the PRYSTINE Traffic Sign Classification by Using Perturbation-based Explainability Approach

Model understanding is critical in many domains, particularly those involved in high-stakes decisions, i.e., medicine, criminal justice, and autonomous driving. Explainable AI (XAI) methods are essential for working with black-box models such as Convolutional Neural Networks. This paper evaluates the traffic sign classifier of Deep Neural Network (DNN) from the Programmable Systems for Intelligence in Automobiles (PRYSTINE) project for explainability. The results of explanations were further used for the CNN PRYSTINE classifier vague kernels` compression. After all, the precision of the classifier was evaluated in different pruning scenarios. The proposed classifier performance methodology was realised by creating the original traffic sign and traffic light classification and explanation code. First, the status of the kernels of the network was evaluated for explainability. For this task, the post-hoc, local, meaningful perturbation-based forward explainable method was integrated into the model to evaluate each kernel status of the network. This method enabled distinguishing high and low-impact kernels in the CNN. Second, the vague kernels of the classifier of the last layer before the fully connected layer were excluded by withdrawing them from the network. Third, the network's precision was evaluated in different kernel compression levels. It is shown that by using the XAI approach for network kernel compression, the pruning of 5% of kernels leads only to a 1% loss in traffic sign and traffic light classification precision. The proposed methodology is crucial where execution time and processing capacity prevail.

Context-Aware and Class Imbalance Invariant Threat Severity Assessment for Heterogeneous IoT

<div>Due to demand for information ubiquity and large-scale automation, proliferating Internet-connected heterogeneous devices exhibit significant variations in data processing capacities, purposes, operating principles, underlying protocols, and dynamic contexts. As a result, adversarial entities exploit the increasing heterogeneous network (HetIoT) vulnerabilities, leading to frequent high-impact attacks due to anomalous device interactions and scarce knowledgebase. This paper presents a two-fold solution to the problem through a network intrusion detection and prevention framework for HetIoT, called \textit{HetIoT-NIDPS}. Firstly, we assign fault scores to the Expert-curated Knowledgebase (EK) framework, correlating with low-level alerts to assess threat severity and achieve context-awareness. Secondly, the proposed Beta distribution-based HetIoT traffic behavior approximation facilitates class imbalance invariance and improves classifier performance. Additionally, the HetIoT-NIDPS can detect zero-day attacks by identifying known attack variations upon encountering unseen traffic instances. Furthermore, the dynamic HetIoT contexts necessitate real-time threat assessment through online training---performed by analyzing small batches of network traffic samples. We propound the \textit{CorrELM} classifier based on the extreme learning machine algorithm and test the hypotheses on the Bot-IoT dataset. Finally, we prioritize the correlated alerts based on their severity, determined from root cause analysis and threat severity assessment tables. The results obtained prove that the proposed HetIoT-NIDPS framework is context-aware---producing reduced false alerts, class imbalance invariant---facilitating near real-time threat assessment with unbiased classifier performance, and generalizable---applicable to many NID datasets, which the existing techniques lack when combined.</div>

Context-Aware and Class Imbalance Invariant Threat Severity Assessment for Heterogeneous IoT

<div>Due to demand for information ubiquity and large-scale automation, proliferating Internet-connected heterogeneous devices exhibit significant variations in data processing capacities, purposes, operating principles, underlying protocols, and dynamic contexts. As a result, adversarial entities exploit the increasing heterogeneous network (HetIoT) vulnerabilities, leading to frequent high-impact attacks due to anomalous device interactions and scarce knowledgebase. This paper presents a two-fold solution to the problem through a network intrusion detection and prevention framework for HetIoT, called \textit{HetIoT-NIDPS}. Firstly, we assign fault scores to the Expert-curated Knowledgebase (EK) framework, correlating with low-level alerts to assess threat severity and achieve context-awareness. Secondly, the proposed Beta distribution-based HetIoT traffic behavior approximation facilitates class imbalance invariance and improves classifier performance. Additionally, the HetIoT-NIDPS can detect zero-day attacks by identifying known attack variations upon encountering unseen traffic instances. Furthermore, the dynamic HetIoT contexts necessitate real-time threat assessment through online training---performed by analyzing small batches of network traffic samples. We propound the \textit{CorrELM} classifier based on the extreme learning machine algorithm and test the hypotheses on the Bot-IoT dataset. Finally, we prioritize the correlated alerts based on their severity, determined from root cause analysis and threat severity assessment tables. The results obtained prove that the proposed HetIoT-NIDPS framework is context-aware---producing reduced false alerts, class imbalance invariant---facilitating near real-time threat assessment with unbiased classifier performance, and generalizable---applicable to many NID datasets, which the existing techniques lack when combined.</div>

Predicting Overall Survival Time in Glioblastoma Patients Using Gradient Boosting Machines Algorithm and Recursive Feature Elimination Technique

Despite advances in tumor treatment, the inconsistent response is a major challenge among glioblastoma multiform (GBM) that lead to different survival time. Our aim was to integrate multimodal MRI with non-supervised and supervised machine learning methods to predict GBM patients’ survival time. To this end, we identified different compartments of the tumor and extracted their features. Next, we applied Random Forest-Recursive Feature Elimination (RF-RFE) to identify the most relevant features to feed into a GBoost machine. This study included 29 GBM patients with known survival time. RF-RFE GBoost model was evaluated to assess the survival prediction performance using optimal features. Furthermore, overall survival (OS) was analyzed using univariate and multivariate Cox regression analyses, to evaluate the effect of ROIs and their features on survival. The results showed that a RF-RFE Gboost machine was able to predict survival time with 75% accuracy. The results also revealed that the rCBV in the low perfusion area was significantly different between groups and had the greatest effect size in terms of the rate of change of the response variable (survival time). In conclusion, not only integration of multi-modality MRI but also feature selection method can enhance the classifier performance.