scholarly journals Transformer-CNN: Fast and Reliable Tool for QSAR

2019 ◽  
Author(s):  
Pavel Karpov ◽  
Guillaume Godin ◽  
Igor Tetko
Keyword(s):  
Transfer Learning ◽  
Source Code ◽  
Future Directions ◽  
Reliable Tool ◽  
Benchmark Datasets ◽  
On Line ◽  
Classification Tasks ◽  
Transformer Model

We present SMILES-embeddings derived from internal encoder state of a Transformer model trained to canonize SMILES as a Seq2Seq problem. Using CharNN architecture upon the embeddings results in a higher quality QSAR/QSPR models on diverse benchmark datasets including regression and classification tasks. The proposed Transformer-CNN method uses SMILES augmentation for training and inference, and thus the prognosis grounds on an internal consensus. Both the augmentation and transfer learning based on embedding allows the method to provide good results for small datasets. We discuss the reasons for such effectiveness and draft future directions for the development of the method. The source code and the embeddings are available on https://github.com/bigchem/transformer-cnn, whereas the OCHEM environment (https://ochem.eu) hosts its on-line implementation.

scholarly journals Transformer-CNN: Fast and Reliable Tool for QSAR

2019 ◽  
Author(s):  
Pavel Karpov ◽  
Guillaume Godin ◽  
Igor Tetko
Keyword(s):  
Transfer Learning ◽  
Source Code ◽  
Future Directions ◽  
Reliable Tool ◽  
Benchmark Datasets ◽  
On Line ◽  
Classification Tasks ◽  
Transformer Model

We present SMILES-embeddings derived from internal encoder state of a Transformer model trained to canonize SMILES as a Seq2Seq problem. Using CharNN architecture upon the embeddings results in a higher quality QSAR/QSPR models on diverse benchmark datasets including regression and classification tasks. The proposed Transformer-CNN method uses SMILES augmentation for training and inference, and thus the prognosis grounds on an internal consensus. Both the augmentation and transfer learning based on embedding allows the method to provide good results for small datasets. We discuss the reasons for such effectiveness and draft future directions for the development of the method. The source code and the embeddings are available on https://github.com/bigchem/transformer-cnn, whereas the OCHEM environment (https://ochem.eu) hosts its on-line implementation.

scholarly journals Discriminative non-negative representation based classifier for image recognition

2021 ◽  
Vol 15 ◽  
pp. 174830262110449
Author(s):  
Kai-Jun Hu ◽  
He-Feng Yin ◽  
Jun Sun
Keyword(s):  
Pattern Classification ◽  
State Of The Art ◽  
Source Code ◽  
Classification Performance ◽  
Training Data ◽  
Practical Applications ◽  
The Past ◽  
Benchmark Datasets ◽  
Classification Tasks ◽  
Negative Representation

During the past decade, representation based classification method has received considerable attention in the community of pattern recognition. The recently proposed non-negative representation based classifier achieved superb recognition results in diverse pattern classification tasks. Unfortunately, discriminative information of training data is not fully exploited in non-negative representation based classifier, which undermines its classification performance in practical applications. To address this problem, we introduce a decorrelation regularizer into the formulation of non-negative representation based classifier and propose a discriminative non-negative representation based classifier for pattern classification. The decorrelation regularizer is able to reduce the correlation of representation results of different classes, thus promoting the competition among them. Experimental results on benchmark datasets validate the efficacy of the proposed discriminative non-negative representation based classifier, and it can outperform some state-of-the-art deep learning based methods. The source code of our proposed discriminative non-negative representation based classifier is accessible at https://github.com/yinhefeng/DNRC .

scholarly journals TPOT-NN: augmenting tree-based automated machine learning with neural network estimators

2021 ◽  
Author(s):  
Joseph D. Romano ◽  
Trang T. Le ◽  
Weixuan Fu ◽  
Jason H. Moore
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Binary Classification ◽  
Inductive Learning ◽  
Future Directions ◽  
High Performing ◽  
Learning Tasks ◽  
Benchmark Datasets ◽  
Automated Machine Learning ◽  
Standard Tree

AbstractAutomated machine learning (AutoML) and artificial neural networks (ANNs) have revolutionized the field of artificial intelligence by yielding incredibly high-performing models to solve a myriad of inductive learning tasks. In spite of their successes, little guidance exists on when to use one versus the other. Furthermore, relatively few tools exist that allow the integration of both AutoML and ANNs in the same analysis to yield results combining both of their strengths. Here, we present TPOT-NN—a new extension to the tree-based AutoML software TPOT—and use it to explore the behavior of automated machine learning augmented with neural network estimators (AutoML+NN), particularly when compared to non-NN AutoML in the context of simple binary classification on a number of public benchmark datasets. Our observations suggest that TPOT-NN is an effective tool that achieves greater classification accuracy than standard tree-based AutoML on some datasets, with no loss in accuracy on others. We also provide preliminary guidelines for performing AutoML+NN analyses, and recommend possible future directions for AutoML+NN methods research, especially in the context of TPOT.

Epileptic Seizure Prediction Using Deep Transformer Model

2021 ◽  
Author(s):  
Abhijeet Bhattacharya ◽  
Tanmay Baweja ◽  
S. P. K. Karri
Keyword(s):  
Signal Processing ◽  
Deep Learning ◽  
False Positive Rate ◽  
Superior Performance ◽  
Seizure Prediction ◽  
Advantages And Disadvantages ◽  
Positive Rate ◽  
Benchmark Datasets ◽  
Automated Screening ◽  
Transformer Model

The electroencephalogram (EEG) is the most promising and efficient technique to study epilepsy and record all the electrical activity going in our brain. Automated screening of epilepsy through data-driven algorithms reduces the manual workload of doctors to diagnose epilepsy. New algorithms are biased either towards signal processing or deep learning, which holds subjective advantages and disadvantages. The proposed pipeline is an end-to-end automated seizure prediction framework with a Fourier transform feature extraction and deep learning-based transformer model, a blend of signal processing and deep learning — this imbibes the potential features to automatically identify the attentive regions in EEG signals for effective screening. The proposed pipeline has demonstrated superior performance on the benchmark dataset with average sensitivity and false-positive rate per hour (FPR/h) as 98.46%, 94.83% and 0.12439, 0, respectively. The proposed work shows great results on the benchmark datasets and a big potential for clinics as a support system with medical experts monitoring the patients.

scholarly journals Instance Enhancement Batch Normalization: An Adaptive Regulator of Batch Noise

2020 ◽  
Vol 34 (04) ◽  
pp. 4819-4827
Author(s):  
Senwei Liang ◽  
Zhongzhan Huang ◽  
Mingfu Liang ◽  
Haizhao Yang
Keyword(s):  
Input Image ◽  
Point Of View ◽  
Specific Information ◽  
Network Training ◽  
Batch Normalization ◽  
Benchmark Datasets ◽  
Classification Tasks ◽  
Adaptive Regulator ◽  
Batch Noise ◽  
Parameter Increment

Batch Normalization (BN) (Ioffe and Szegedy 2015) normalizes the features of an input image via statistics of a batch of images and hence BN will bring the noise to the gradient of training loss. Previous works indicate that the noise is important for the optimization and generalization of deep neural networks, but too much noise will harm the performance of networks. In our paper, we offer a new point of view that the self-attention mechanism can help to regulate the noise by enhancing instance-specific information to obtain a better regularization effect. Therefore, we propose an attention-based BN called Instance Enhancement Batch Normalization (IEBN) that recalibrates the information of each channel by a simple linear transformation. IEBN has a good capacity of regulating the batch noise and stabilizing network training to improve generalization even in the presence of two kinds of noise attacks during training. Finally, IEBN outperforms BN with only a light parameter increment in image classification tasks under different network structures and benchmark datasets.

Deep Learning-Based Classification of Malignant and Benign Cells in Dermatoscopic Images via Transfer Learning Approach

2021 ◽  
Author(s):  
V. Akash Kumar ◽  
Vijaya Mishra ◽  
Monika Arora
Keyword(s):  
Skin Cancer ◽  
Transfer Learning ◽  
Human Life ◽  
Skin Pigmentation ◽  
Automatic Identification ◽  
Learning Approach ◽  
Predicted Model ◽  
Classification Tasks ◽  
Cancerous Cells

The inhibition of healthy cells creating improper controlling process of the human body system indicates the occurrence of growth of cancerous cells. The cluster of such cells leads to the development of tumor. The observation of this type of abnormal skin pigmentation is done using an effective tool called Dermoscopy. However, these dermatoscopic images possess a great challenge for diagnosis. Considering the characteristics of dermatoscopic images, transfer learning is an appropriate approach of automatically classifying the images based on the respective categories. An automatic identification of skin cancer not only saves human life but also helps in detecting its growth at an earlier stage which saves medical practitioner’s effort and time. A newly predicted model has been proposed for classifying the skin cancer as benign or malignant by DCNN with transfer learning and its pre-trained models such as VGG 16, VGG 19, ResNet 50, ResNet 101, and Inception V3. The proposed methodology aims at examining the efficiency of pre-trained models and transfer learning approach for the classification tasks and opens new dimensions of research in the field of medicines using imaging technique which can be implementable in real-time applications.

scholarly journals Explicit Interaction Model towards Text Classification

2019 ◽  
Vol 33 ◽  
pp. 6359-6366 ◽  
Author(s):  
Cunxiao Du ◽  
Zhaozheng Chen ◽  
Fuli Feng ◽  
Lei Zhu ◽  
Tian Gan ◽  
...  
Keyword(s):  
Language Processing ◽  
Text Classification ◽  
Deep Neural Networks ◽  
Interaction Mechanism ◽  
Interaction Model ◽  
Classification Task ◽  
Fine Grained ◽  
Word Level ◽  
Benchmark Datasets ◽  
Classification Tasks

Text classification is one of the fundamental tasks in natural language processing. Recently, deep neural networks have achieved promising performance in the text classification task compared to shallow models. Despite of the significance of deep models, they ignore the fine-grained (matching signals between words and classes) classification clues since their classifications mainly rely on the text-level representations. To address this problem, we introduce the interaction mechanism to incorporate word-level matching signals into the text classification task. In particular, we design a novel framework, EXplicit interAction Model (dubbed as EXAM), equipped with the interaction mechanism. We justified the proposed approach on several benchmark datasets including both multilabel and multi-class text classification tasks. Extensive experimental results demonstrate the superiority of the proposed method. As a byproduct, we have released the codes and parameter settings to facilitate other researches.

Source code-based defect prediction using deep learning and transfer learning

2019 ◽  
Vol 23 (6) ◽  
pp. 1243-1269
Author(s):  
Ahmad A. Saifan ◽  
Nawzat Al Smadi
Keyword(s):  
Deep Learning ◽  
Transfer Learning ◽  
Source Code ◽  
Defect Prediction

scholarly journals Deep Transfer Learning for Source Code Modeling

2020 ◽  
Vol 30 (05) ◽  
pp. 649-668
Author(s):  
Yasir Hussain ◽  
Zhiqiu Huang ◽  
Yu Zhou ◽  
Senzhang Wang
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Transfer Learning ◽  
Related Problem ◽  
State Of The Art ◽  
Source Code ◽  
Traditional Learning ◽  
Learning Models ◽  
Modeling And Analysis ◽  
F Measure

In recent years, deep learning models have shown great potential in source code modeling and analysis. Generally, deep learning-based approaches are problem-specific and data-hungry. A challenging issue of these approaches is that they require training from scratch for a different related problem. In this work, we propose a transfer learning-based approach that significantly improves the performance of deep learning-based source code models. In contrast to traditional learning paradigms, transfer learning can transfer the knowledge learned in solving one problem into another related problem. First, we present two recurrent neural network-based models RNN and GRU for the purpose of transfer learning in the domain of source code modeling. Next, via transfer learning, these pre-trained (RNN and GRU) models are used as feature extractors. Then, these extracted features are combined into attention learner for different downstream tasks. The attention learner leverages from the learned knowledge of pre-trained models and fine-tunes them for a specific downstream task. We evaluate the performance of the proposed approach with extensive experiments with the source code suggestion task. The results indicate that the proposed approach outperforms the state-of-the-art models in terms of accuracy, precision, recall and F-measure without training the models from scratch.

Sign in / Sign up

Export Citation Format

Share Document