Characterization of Symbolic Rules Embedded in Deep DIMLP Networks: A Challenge to Transparency of Deep Learning

AbstractRule extraction from neural networks is a fervent research topic. In the last 20 years many authors presented a number of techniques showing how to extract symbolic rules from Multi Layer Perceptrons (MLPs). Nevertheless, very few were related to ensembles of neural networks and even less for networks trained by deep learning. On several datasets we performed rule extraction from ensembles of Discretized Interpretable Multi Layer Perceptrons (DIMLP), and DIMLPs trained by deep learning. The results obtained on the Thyroid dataset and the Wisconsin Breast Cancer dataset show that the predictive accuracy of the extracted rules compare very favorably with respect to state of the art results. Finally, in the last classification problem on digit recognition, generated rules from the MNIST dataset can be viewed as discriminatory features in particular digit areas. Qualitatively, with respect to rule complexity in terms of number of generated rules and number of antecedents per rule, deep DIMLPs and DIMLPs trained by arcing give similar results on a binary classification problem involving digits 5 and 8. On the whole MNIST problem we showed that it is possible to determine the feature detectors created by neural networks and also that the complexity of the extracted rulesets can be well balanced between accuracy and interpretability.

Download Full-text

Advancing Stress Detection Methodology with Deep Learning Techniques Targeting UX Evaluation in AAL Scenarios: Applying Embeddings for Categorical Variables

Electronics ◽

10.3390/electronics10131550 ◽

2021 ◽

Vol 10 (13) ◽

pp. 1550

Author(s):

Alexandros Liapis ◽

Evanthia Faliagka ◽

Christos P. Antonopoulos ◽

Georgios Keramidas ◽

Nikolaos Voros

Keyword(s):

Machine Learning ◽

Deep Learning ◽

User Experience ◽

Electrodermal Activity ◽

Binary Classification ◽

Research Question ◽

Classification Problem ◽

Categorical Variables ◽

Stress Detection ◽

Software Failures

Physiological measurements have been widely used by researchers and practitioners in order to address the stress detection challenge. So far, various datasets for stress detection have been recorded and are available to the research community for testing and benchmarking. The majority of the stress-related available datasets have been recorded while users were exposed to intense stressors, such as songs, movie clips, major hardware/software failures, image datasets, and gaming scenarios. However, it remains an open research question if such datasets can be used for creating models that will effectively detect stress in different contexts. This paper investigates the performance of the publicly available physiological dataset named WESAD (wearable stress and affect detection) in the context of user experience (UX) evaluation. More specifically, electrodermal activity (EDA) and skin temperature (ST) signals from WESAD were used in order to train three traditional machine learning classifiers and a simple feed forward deep learning artificial neural network combining continues variables and entity embeddings. Regarding the binary classification problem (stress vs. no stress), high accuracy (up to 97.4%), for both training approaches (deep-learning, machine learning), was achieved. Regarding the stress detection effectiveness of the created models in another context, such as user experience (UX) evaluation, the results were quite impressive. More specifically, the deep-learning model achieved a rather high agreement when a user-annotated dataset was used for validation.

Download Full-text

Deep Learning for Laryngopharyngeal Reflux Diagnosis

Applied Sciences ◽

10.3390/app11114753 ◽

2021 ◽

Vol 11 (11) ◽

pp. 4753

Author(s):

Gen Ye ◽

Chen Du ◽

Tong Lin ◽

Yan Yan ◽

Jack Jiang

Keyword(s):

Deep Learning ◽

Speech Processing ◽

Data Augmentation ◽

Laryngopharyngeal Reflux ◽

Ph Monitoring ◽

Binary Classification ◽

Classification Problem ◽

Learning Approaches ◽

Learning Techniques ◽

Auc Value

(1) Background: Deep learning has become ubiquitous due to its impressive performance in various domains, such as varied as computer vision, natural language and speech processing, and game-playing. In this work, we investigated the performance of recent deep learning approaches on the laryngopharyngeal reflux (LPR) diagnosis task. (2) Methods: Our dataset is composed of 114 subjects with 37 pH-positive cases and 77 control cases. In contrast to prior work based on either reflux finding score (RFS) or pH monitoring, we directly take laryngoscope images as inputs to neural networks, as laryngoscopy is the most common and simple diagnostic method. The diagnosis task is formulated as a binary classification problem. We first tested a powerful backbone network that incorporates residual modules, attention mechanism and data augmentation. Furthermore, recent methods in transfer learning and few-shot learning were investigated. (3) Results: On our dataset, the performance is the best test classification accuracy is 73.4%, while the best AUC value is 76.2%. (4) Conclusions: This study demonstrates that deep learning techniques can be applied to classify LPR images automatically. Although the number of pH-positive images used for training is limited, deep network can still be capable of learning discriminant features with the advantage of technique.

Download Full-text

Deep-learning approach to identifying cancer subtypes using high-dimensional genomic data

Bioinformatics ◽

10.1093/bioinformatics/btz769 ◽

2019 ◽

Cited By ~ 3

Author(s):

Runpu Chen ◽

Le Yang ◽

Steve Goodison ◽

Yijun Sun

Keyword(s):

Deep Learning ◽

Cluster Structure ◽

Data Representation ◽

Supplementary Information ◽

High Dimensional ◽

Breast Cancer Dataset ◽

Cancer Dataset ◽

Cancer Subtypes ◽

Novel Approach ◽

Open Source Software Package

Abstract Motivation Cancer subtype classification has the potential to significantly improve disease prognosis and develop individualized patient management. Existing methods are limited by their ability to handle extremely high-dimensional data and by the influence of misleading, irrelevant factors, resulting in ambiguous and overlapping subtypes. Results To address the above issues, we proposed a novel approach to disentangling and eliminating irrelevant factors by leveraging the power of deep learning. Specifically, we designed a deep-learning framework, referred to as DeepType, that performs joint supervised classification, unsupervised clustering and dimensionality reduction to learn cancer-relevant data representation with cluster structure. We applied DeepType to the METABRIC breast cancer dataset and compared its performance to state-of-the-art methods. DeepType significantly outperformed the existing methods, identifying more robust subtypes while using fewer genes. The new approach provides a framework for the derivation of more accurate and robust molecular cancer subtypes by using increasingly complex, multi-source data. Availability and implementation An open-source software package for the proposed method is freely available at http://www.acsu.buffalo.edu/~yijunsun/lab/DeepType.html. Supplementary information Supplementary data are available at Bioinformatics online.

Download Full-text

Utilizing Information Bottleneck to Evaluate the Capability of Deep Neural Networks for Image Classification

Entropy ◽

10.3390/e21050456 ◽

2019 ◽

Vol 21 (5) ◽

pp. 456 ◽

Cited By ~ 3

Author(s):

Hao Cheng ◽

Dongze Lian ◽

Shenghua Gao ◽

Yanlin Geng

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Model Selection ◽

Image Classification ◽

Transfer Learning ◽

Deep Neural Networks ◽

Classification Problem ◽

Distribution Model ◽

Classification Problems ◽

Information Bottleneck

Inspired by the pioneering work of the information bottleneck (IB) principle for Deep Neural Networks’ (DNNs) analysis, we thoroughly study the relationship among the model accuracy, I ( X ; T ) and I ( T ; Y ) , where I ( X ; T ) and I ( T ; Y ) are the mutual information of DNN’s output T with input X and label Y. Then, we design an information plane-based framework to evaluate the capability of DNNs (including CNNs) for image classification. Instead of each hidden layer’s output, our framework focuses on the model output T. We successfully apply our framework to many application scenarios arising in deep learning and image classification problems, such as image classification with unbalanced data distribution, model selection, and transfer learning. The experimental results verify the effectiveness of the information plane-based framework: Our framework may facilitate a quick model selection and determine the number of samples needed for each class in the unbalanced classification problem. Furthermore, the framework explains the efficiency of transfer learning in the deep learning area.

Download Full-text

Optimal weighted feature vector and deep belief network for medical data classification

International Journal of Wavelets Multiresolution and Information Processing ◽

10.1142/s021969132050006x ◽

2019 ◽

Vol 18 (02) ◽

pp. 2050006 ◽

Cited By ~ 1

Author(s):

Vinod Jagannath Kadam ◽

Shivajirao Manikrao Jadhav

Keyword(s):

Deep Learning ◽

Data Classification ◽

Medical Data ◽

Deep Belief Network ◽

Breast Cancer Dataset ◽

Cancer Dataset ◽

Belief Network ◽

Medical Diagnoses ◽

Whale Optimization ◽

Medical Data Classification

Medical data classification is the process of transforming descriptions of medical diagnoses and procedures into universal medical code numbers. The diagnoses and procedures are usually taken from a variety of sources within the healthcare record, such as the transcription of the physician’s notes, laboratory results, radiologic results and other sources. However, there exist many frequency distribution problems in these domains. Hence, this paper intends to develop an advanced and precise medical data classification approach for diabetes and breast cancer dataset. With the knowledge of the features and challenges persisting with the state-of-the-art classification methods, deep learning-based medical data classification methodology is proposed here. It is well known that deep learning networks learn directly from the data. In this paper, the medical data is dimensionally reduced using Principle Component Analysis (PCA). The dimensionally reduced data are transformed by multiplying by a weighting factor, which is optimized using Whale Optimization Algorithm (WOA), to obtain the maximum distance between the features. As a result, the data are transformed into a label-distinguishable plane under which the Deep Belief Network (DBN) is adopted to perform the deep learning process, and the data classification is performed. Further, the proposed WOA-based DBN (WOADBN) method is compared with the Neural Network (NN), DBN, Generic Algorithm-based NN (GANN), GADBN, Particle Swarm Optimization (PSONN), PSO-based DBN (PSODBN), WOA-based NN (WOANN) techniques and the results are obtained, which shows the superiority of proposed algorithm over conventional methods.

Download Full-text

PM2.5 Prediction Model Based on Combinational Hammerstein Recurrent Neural Networks

Mathematics ◽

10.3390/math8122178 ◽

2020 ◽

Vol 8 (12) ◽

pp. 2178

Author(s):

Yi-Chung Chen ◽

Tsu-Chiang Lei ◽

Shun Yao ◽

Hsin-Ping Wang

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Recurrent Neural Networks ◽

Predictive Accuracy ◽

Learning Models ◽

Short Term ◽

Computational Overhead ◽

Implementation Costs ◽

External Sources

Airborne particulate matter 2.5 (PM2.5) can have a profound effect on the health of the population. Many researchers have been reporting highly accurate numerical predictions based on raw PM2.5 data imported directly into deep learning models; however, there is still considerable room for improvement in terms of implementation costs due to heavy computational overhead. From the perspective of environmental science, PM2.5 values in a given location can be attributed to local sources as well as external sources. Local sources tend to have a dramatic short-term impact on PM2.5 values, whereas external sources tend to have more subtle but longer-lasting effects. In the presence of PM2.5 from both sources at the same time, this combination of effects can undermine the predictive accuracy of the model. This paper presents a novel combinational Hammerstein recurrent neural network (CHRNN) to enhance predictive accuracy and overcome the heavy computational and monetary burden imposed by deep learning models. The CHRNN comprises a based-neural network tasked with learning gradual (long-term) fluctuations in conjunction with add-on neural networks to deal with dramatic (short-term) fluctuations. The CHRNN can be coupled with a random forest model to determine the degree to which short-term effects influence long-term outcomes. We also developed novel feature selection and normalization methods to enhance prediction accuracy. Using real-world measurement data of air quality and PM2.5 datasets from Taiwan, the precision of the proposed system in the numerical prediction of PM2.5 levels was comparable to that of state-of-the-art deep learning models, such as deep recurrent neural networks and long short-term memory, despite far lower implementation costs and computational overhead.

Download Full-text

Tens of images can suffice to train neural networks for malignant leukocyte detection

Scientific Reports ◽

10.1038/s41598-021-86995-5 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Jens P. E. Schouten ◽

Christian Matek ◽

Luuk F. P. Jacobs ◽

Michèle C. Buck ◽

Dragan Bošnački ◽

...

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Single Cell ◽

Expert Knowledge ◽

Binary Classification ◽

Lymphoblastic Leukemia ◽

Training Set ◽

Early Detection Of Disease ◽

Set Size ◽

Training Images

AbstractConvolutional neural networks (CNNs) excel as powerful tools for biomedical image classification. It is commonly assumed that training CNNs requires large amounts of annotated data. This is a bottleneck in many medical applications where annotation relies on expert knowledge. Here, we analyze the binary classification performance of a CNN on two independent cytomorphology datasets as a function of training set size. Specifically, we train a sequential model to discriminate non-malignant leukocytes from blast cells, whose appearance in the peripheral blood is a hallmark of leukemia. We systematically vary training set size, finding that tens of training images suffice for a binary classification with an ROC-AUC over 90%. Saliency maps and layer-wise relevance propagation visualizations suggest that the network learns to increasingly focus on nuclear structures of leukocytes as the number of training images is increased. A low dimensional tSNE representation reveals that while the two classes are separated already for a few training images, the distinction between the classes becomes clearer when more training images are used. To evaluate the performance in a multi-class problem, we annotated single-cell images from a acute lymphoblastic leukemia dataset into six different hematopoietic classes. Multi-class prediction suggests that also here few single-cell images suffice if differences between morphological classes are large enough. The incorporation of deep learning algorithms into clinical practice has the potential to reduce variability and cost, democratize usage of expertise, and allow for early detection of disease onset and relapse. Our approach evaluates the performance of a deep learning based cytology classifier with respect to size and complexity of the training data and the classification task.

Download Full-text

Deep Convolution Neural Network Based Breast Cancer Bigdata Analysis for Crowd Cloud Sourcing

International Journal of Innovative Technology and Exploring Engineering - Special Issue ◽

10.35940/ijitee.c1081.0193s20 ◽

2020 ◽

Vol 9 (3S) ◽

pp. 378-383

Keyword(s):

Breast Cancer ◽

Deep Learning ◽

Learning Algorithms ◽

Medical Industry ◽

Machine Learning Algorithms ◽

The Other ◽

Major Surgery ◽

Breast Cancer Dataset ◽

Cancer Dataset ◽

Deep Convolution Neural Network

Breast cancer is one of the dangerous diseases leads fast death among women. Several kinds of cancers are affecting people, but breast cancer affects highly women. In medical industry removal of women breasts or major surgery is taken forward as the solution, where it reoccurs after surgery also. Only solution to save women from breast cancer is to identify and detect the earlier stage of cancer and provide necessary treatment. Hence various research works have been focused on finding good solution for diagnosing and classifying the cancer stages as benign, malignant or severe malignant. Still the accuracy of classification needs to be improved on complex breast cancer datasets. Few of the earlier research works have proposed machine learning algorithms, which are semiautomatic and accuracy is also not high. Thus, to provide a better solution this paper aimed to use one of the deep learning algorithms such as Convolution Neural Networks for diagnosing various kinds of breast cancer dataset. From the experimental results, it is obtained that the proposed deep learning algorithms outperforms than the other algorithms.

Download Full-text

ABCnet : Self-Attention based Atom, Bond Message Passing Network for Predicting Drug-Target Interaction

10.1101/2021.12.27.474154 ◽

2021 ◽

Author(s):

Segyu Lee ◽

Junil Bang ◽

Sungeun Hong ◽

Woojung Jang

Keyword(s):

Deep Learning ◽

Drug Development ◽

Binding Affinity ◽

Message Passing ◽

Drug Target ◽

Binary Classification ◽

Classification Problem ◽

Molecular Structures ◽

Target Interaction ◽

Molecular Features

Drug-target interaction (DTI) is a methodology for predicting the binding affinity between a compound and a target protein, and a key technology in the derivation of candidate substances in drug discovery. As DTI experiments have progressed for a long time, a substantial volume of chemical, biomedical, and pharmaceutical data have accumulated. This accumulation of data has occurred contemporaneously with the advent of the field of big data, and data-based machine learning methods could significantly reduce the time and cost of drug development. In particular, the deep learning method shows potential when applied to the fields of vision and speech recognition, and studies to apply deep learning to various other fields have emerged. Research applying deep learning is underway in drug development, and among various deep learning models, a graph-based model that can effectively learn molecular structures has received more attention as the SOTA in experimental results were achieved. Our study focused on molecular structure information among graph-based models in message passing neural networks. In this paper, we propose a self-attention-based bond and atom message passing neural network which predicts DTI by extracting molecular features through a graph model using an attention mechanism. Model validation experiments were performed after defining binding affinity as a regression and classification problem: binary classification to predict the presence or absence of binding to the drug-target, and regression to predict binding affinity to the drug-target. Classification was performed with BindingDB, and regression was performed with the DAVIS dataset. In the classification problem, ABCnet showed higher performance than MPNN, as it does in the existing study, and in regression, the potential of ABCnet was checked compared to that of SOTA. Experiments indicated that in binary classification, ABCnet has an average performance improvement of 1% than other MPNN on the DTI task, and in regression, ABCnet has CI and performance degradation between 0.01 and 0.02 compared to SOTA.

Download Full-text

METHODS FOR INCREASING THE CLASSIFICATION ACCURACY BASED ON MODIFICATIONS OF THE BASIC ARCHITECTURE OF CONVOLUTIONAL NEURAL NETWORKS

ScienceRise ◽

10.21303/2313-8416.2020.001550 ◽

2020 ◽

pp. 10-16

Author(s):

Svitlana Shapovalova ◽

Yurii Moskalenko

Keyword(s):

Neural Networks ◽

Deep Learning ◽

Classification Accuracy ◽

Malignant Tumors ◽

Classification Problem ◽

Video Memory ◽

Backbone Networks ◽

Convolutional Networks ◽

High Recognition Accuracy ◽

Technological Product

Object of research: basic architectures of deep learning neural networks. Investigated problem: insufficient accuracy of solving the classification problem based on the basic architectures of deep learning neural networks. An increase in accuracy requires a significant complication of the architecture, which, in turn, leads to an increase in the required computing resources, as well as the consumption of video memory and the cost of learning/output time. Therefore, the problem arises of determining such methods for modifying basic architectures that improve the classification accuracy and require insignificant additional computing resources. Main scientific results: based on the analysis of existing methods for improving the classification accuracy on the convolutional networks of basic architectures, it is determined what is most effective: scaling the ScanNet architecture, learning the ensemble of TreeNet models, integrating several CBNet backbone networks. For computational experiments, these modifications of the basic architectures are implemented, as well as their combinations: ScanNet + TreeNet, ScanNet + CBNet. The effectiveness of these methods in comparison with basic architectures has been proven when solving the problem of recognizing malignant tumors with diagnostic images – SIIM-ISIC Melanoma Classification, the train/test set of which is presented on the Kaggle platform. The accuracy value for the area under the ROC curve metric has increased from 0.94489 (basic architecture network) to 0.96317 (network with ScanNet + CBNet modifications). At the same time, the output compared to the basic architecture (EfficientNet-b5) increased from 440 to 490 seconds, and the consumption of video memory increased from 8 to 9.2 gigabytes, which is acceptable. Innovative technological product: methods for achieving high recognition accuracy from a diagnostic signal based on deep learning neural networks of basic architectures. Scope of application of the innovative technological product: automatic diagnostics systems in the following areas: medicine, seismology, astronomy (classification by images) onboard control systems and systems for monitoring transport and vehicle flows or visitors (recognition of scenes with camera frames).

Download Full-text