What You See is What it Means! Semantic Representation Learning of Code based on Visualization and Transfer Learning

Recent successes in training word embeddings for Natural Language Processing ( NLP ) tasks have encouraged a wave of research on representation learning for source code, which builds on similar NLP methods. The overall objective is then to produce code embeddings that capture the maximum of program semantics. State-of-the-art approaches invariably rely on a syntactic representation (i.e., raw lexical tokens, abstract syntax trees, or intermediate representation tokens) to generate embeddings, which are criticized in the literature as non-robust or non-generalizable. In this work, we investigate a novel embedding approach based on the intuition that source code has visual patterns of semantics. We further use these patterns to address the outstanding challenge of identifying semantic code clones. We propose the WySiWiM ( ‘ ‘What You See Is What It Means ” ) approach where visual representations of source code are fed into powerful pre-trained image classification neural networks from the field of computer vision to benefit from the practical advantages of transfer learning. We evaluate the proposed embedding approach on the task of vulnerable code prediction in source code and on two variations of the task of semantic code clone identification: code clone detection (a binary classification problem), and code classification (a multi-classification problem). We show with experiments on the BigCloneBench (Java), Open Judge (C) that although simple, our WySiWiM approach performs as effectively as state-of-the-art approaches such as ASTNN or TBCNN. We also showed with data from NVD and SARD that WySiWiM representation can be used to learn a vulnerable code detector with reasonable performance (accuracy ∼90%). We further explore the influence of different steps in our approach, such as the choice of visual representations or the classification algorithm, to eventually discuss the promises and limitations of this research direction.

Download Full-text

Deep Transfer Learning for Source Code Modeling

International Journal of Software Engineering and Knowledge Engineering ◽

10.1142/s0218194020500230 ◽

2020 ◽

Vol 30 (05) ◽

pp. 649-668

Author(s):

Yasir Hussain ◽

Zhiqiu Huang ◽

Yu Zhou ◽

Senzhang Wang

Keyword(s):

Neural Network ◽

Deep Learning ◽

Transfer Learning ◽

How Different Text-Preprocessing Techniques using the Bert Model Affect the Gender Profiling of Authors

10.5121/csit.2021.111501 ◽

2021 ◽

Author(s):

Esam Alzahrani ◽

Leon Jololian

Keyword(s):

Transfer Learning ◽

Language Processing ◽

Significant Variation ◽

State Of The Art ◽

Text Preprocessing ◽

The Many ◽

Preprocessing Technique ◽

Multiple References ◽

Author Profiling ◽

Model Affect

Forensic author profiling plays an important role in indicating possible profiles for suspects. Among the many automated solutions recently proposed for author profiling, transfer learning outperforms many other state-of-the-art techniques in natural language processing. Nevertheless, the sophisticated technique has yet to be fully exploited for author profiling. At the same time, whereas current methods of author profiling, all largely based on features engineering, have spawned significant variation in each model used, transfer learning usually requires a preprocessed text to be fed into the model. We reviewed multiple references in the literature and determined the most common preprocessing techniques associated with authors' genders profiling. Considering the variations in potential preprocessing techniques, we conducted an experimental study that involved applying five such techniques to measure each technique’s effect while using the BERT model, chosen for being one of the most-used stock pretrained models. We used the Hugging face transformer library to implement the code for each preprocessing case. In our five experiments, we found that BERT achieves the best accuracy in predicting the gender of the author when no preprocessing technique is applied. Our best case achieved 86.67% accuracy in predicting the gender of authors.

Download Full-text

COREFERENT PAIRS DETECTION IN UKRAINIAN TEXTS USING A CONVOLUTIONAL NEURAL NETWORK

Visnyk Universytetu “Ukraina” ◽

10.36994/2707-4110-2019-2-23-25 ◽

2019 ◽

Author(s):

Sergiy Pogorilyy ◽

Artem Kramov

Keyword(s):

Neural Network ◽

Machine Learning ◽

Convolutional Neural Network ◽

Language Processing ◽

State Of The Art ◽

Classification Problem ◽

Machine Learning Algorithms ◽

Coreference Resolution ◽

Training Process ◽

Natural Languages

The detection of coreferent pairs within a text is one of the basic tasks in the area of natural language processing (NLP). The state‑ of‑ the‑ art methods of coreference resolution are based on machine learning algorithms. The key idea of the methods is to detect certain regularities between the semantic or grammatical features of text entities. In the paper, the comparative analysis of current methods of coreference resolution in English and Ukrainian texts has been performed. The key disadvantage of many methods consists in the interpretation of coreference resolution as a classification problem. The result of coreferent pairs detection is the set of groups in which elements refer to a common entity. Therefore it is advisable to consider the coreference resolution as a clusterization task. The method of coreference resolution using the set of filtering sieves and a convolutional neural network has been suggested. The set of filtering sieves to find candidates for coreferent pairs formation has been implemented. The training process of a multichannel convolutional neural network on a marked Ukrainian corpus has been performed. The usage of a multichannel structure allows analyzing of the different components of text units: semantic, lexical, and grammatical features of words and sentences. Furthermore, it is possible to process input data with unfixed size (words or sentences of a text) using a convolutional layer. The output result of the method is the set of clusters. In order to form clusters, it is necessary to take into account the previous steps of the model’s workflow. Nevertheless, such an approach contradicts the traditional methodology of machine learning. Thus, the training process of the network has been performed using the SEARN algorithm that allows the solving of tasks with unfixed output structures using a classifier model. An experimental examination of the method on the corpus of Ukrainian news has been performed. In order to estimate the accuracy of the method the corresponding common metrics for clusterization tasks have been calculated. The results obtained can indicate that the suggested method can be used to find coreferent pairs within Ukrainian texts. The method can be also easily adapted and applied to other natural languages.

Download Full-text

Detecting operons in bacterial genomes via visual representation learning

Scientific Reports ◽

10.1038/s41598-021-81169-9 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Rida Assaf ◽

Fangfang Xia ◽

Rick Stevens

Keyword(s):

Regulatory Networks ◽

Data Augmentation ◽

State Of The Art ◽

Critical Role ◽

Visual Representations ◽

Representation Learning ◽

Genomic Region ◽

Bacterial Genomes ◽

Gene Neighborhood ◽

Augmentation Techniques

AbstractContiguous genes in prokaryotes are often arranged into operons. Detecting operons plays a critical role in inferring gene functionality and regulatory networks. Human experts annotate operons by visually inspecting gene neighborhoods across pileups of related genomes. These visual representations capture the inter-genic distance, strand direction, gene size, functional relatedness, and gene neighborhood conservation, which are the most prominent operon features mentioned in the literature. By studying these features, an expert can then decide whether a genomic region is part of an operon. We propose a deep learning based method named Operon Hunter that uses visual representations of genomic fragments to make operon predictions. Using transfer learning and data augmentation techniques facilitates leveraging the powerful neural networks trained on image datasets by re-training them on a more limited dataset of extensively validated operons. Our method outperforms the previously reported state-of-the-art tools, especially when it comes to predicting full operons and their boundaries accurately. Furthermore, our approach makes it possible to visually identify the features influencing the network’s decisions to be subsequently cross-checked by human experts.

Download Full-text

MolRep: A Deep Representation Learning Library for Molecular Property Prediction

10.1101/2021.01.13.426489 ◽

2021 ◽

Author(s):

Jiahua Rao ◽

Shuangjia Zheng ◽

Ying Song ◽

Jianwen Chen ◽

Chengtao Li ◽

...

Keyword(s):

State Of The Art ◽

Source Code ◽

Representation Learning ◽

Supplementary Information ◽

Data Sets ◽

Supplementary Data ◽

Property Prediction ◽

Average Rank ◽

Benchmark Data ◽

Classification Tasks

AbstractSummaryRecently, novel representation learning algorithms have shown potential for predicting molecular properties. However, unified frameworks have not yet emerged for fairly measuring algorithmic progress, and experimental procedures of different representation models often lack rigorousness and are hardly reproducible. Herein, we have developed MolRep by unifying 16 state-of-the-art models across 4 popular molecular representations for application and comparison. Furthermore, we ran more than 12.5 million experiments to optimize hyperparameters for each method on 12 common benchmark data sets. As a result, CMPNN achieves the best results ranked the 1st in 5 out of 12 tasks with an average rank of 1.75. Relatively, ECC has good performance in classification tasks and MAT good for regression (both ranked 1st for 3 tasks) with an average rank of 2.71 and 2.6, respectively.AvailabilityThe source code is available at: https://github.com/biomed-AI/MolRepSupplementary informationSupplementary data are available online.

Download Full-text

Active Learning for Effectively Fine-Tuning Transfer Learning to Downstream Task

ACM Transactions on Intelligent Systems and Technology ◽

10.1145/3446343 ◽

2021 ◽

Vol 12 (2) ◽

pp. 1-24

Author(s):

Md Abul Bashar ◽

Richi Nayak

Keyword(s):

Active Learning ◽

Transfer Learning ◽

Language Processing ◽

State Of The Art ◽

Language Model ◽

Ensemble Classifier ◽

Classification Performance ◽

Fine Tuning ◽

Linguistic Features ◽

Better Than

Language model (LM) has become a common method of transfer learning in Natural Language Processing (NLP) tasks when working with small labeled datasets. An LM is pretrained using an easily available large unlabelled text corpus and is fine-tuned with the labelled data to apply to the target (i.e., downstream) task. As an LM is designed to capture the linguistic aspects of semantics, it can be biased to linguistic features. We argue that exposing an LM model during fine-tuning to instances that capture diverse semantic aspects (e.g., topical, linguistic, semantic relations) present in the dataset will improve its performance on the underlying task. We propose a Mixed Aspect Sampling (MAS) framework to sample instances that capture different semantic aspects of the dataset and use the ensemble classifier to improve the classification performance. Experimental results show that MAS performs better than random sampling as well as the state-of-the-art active learning models to abuse detection tasks where it is hard to collect the labelled data for building an accurate classifier.

Download Full-text

Dependency parsing of biomedical text with BERT

BMC Bioinformatics ◽

10.1186/s12859-020-03905-8 ◽

2020 ◽

Vol 21 (S23) ◽

Author(s):

Jenna Kanerva ◽

Filip Ginter ◽

Sampo Pyysalo

Keyword(s):

Transfer Learning ◽

Language Processing ◽

State Of The Art ◽

Text Processing ◽

Syntactic Analysis ◽

Biomedical Text ◽

Dependency Parsing ◽

Shared Task ◽

Fine Tune ◽

Selection Of

Abstract Background: Syntactic analysis, or parsing, is a key task in natural language processing and a required component for many text mining approaches. In recent years, Universal Dependencies (UD) has emerged as the leading formalism for dependency parsing. While a number of recent tasks centering on UD have substantially advanced the state of the art in multilingual parsing, there has been only little study of parsing texts from specialized domains such as biomedicine. Methods: We explore the application of state-of-the-art neural dependency parsing methods to biomedical text using the recently introduced CRAFT-SA shared task dataset. The CRAFT-SA task broadly follows the UD representation and recent UD task conventions, allowing us to fine-tune the UD-compatible Turku Neural Parser and UDify neural parsers to the task. We further evaluate the effect of transfer learning using a broad selection of BERT models, including several models pre-trained specifically for biomedical text processing. Results: We find that recently introduced neural parsing technology is capable of generating highly accurate analyses of biomedical text, substantially improving on the best performance reported in the original CRAFT-SA shared task. We also find that initialization using a deep transfer learning model pre-trained on in-domain texts is key to maximizing the performance of the parsing methods.

Download Full-text

Unsupervised extractive multi-document summarization method based on transfer learning from BERT multi-task fine-tuning

Journal of Information Science ◽

10.1177/0165551521990616 ◽

2021 ◽

pp. 016555152199061

Author(s):

Salima Lamsiyah ◽

Abdelkader El Mahdaouy ◽

Saïd El Alaoui Ouatik ◽

Bernard Espinasse

Keyword(s):

Transfer Learning ◽

State Of The Art ◽

Representation Learning ◽

Fine Tuning ◽

Text Representation ◽

Document Summarization ◽

Semantic Relationships ◽

Benchmark Datasets ◽

Tuning Methods ◽

Fine Tune

Text representation is a fundamental cornerstone that impacts the effectiveness of several text summarization methods. Transfer learning using pre-trained word embedding models has shown promising results. However, most of these representations do not consider the order and the semantic relationships between words in a sentence, and thus they do not carry the meaning of a full sentence. To overcome this issue, the current study proposes an unsupervised method for extractive multi-document summarization based on transfer learning from BERT sentence embedding model. Moreover, to improve sentence representation learning, we fine-tune BERT model on supervised intermediate tasks from GLUE benchmark datasets using single-task and multi-task fine-tuning methods. Experiments are performed on the standard DUC’2002–2004 datasets. The obtained results show that our method has significantly outperformed several baseline methods and achieves a comparable and sometimes better performance than the recent state-of-the-art deep learning–based methods. Furthermore, the results show that fine-tuning BERT using multi-task learning has considerably improved the performance.

Download Full-text

TRS: Transformers for Remote Sensing Scene Classification

Remote Sensing ◽

10.3390/rs13204143 ◽

2021 ◽

Vol 13 (20) ◽

pp. 4143

Author(s):

Jianrong Zhang ◽

Hongwei Zhao ◽

Jiao Li

Keyword(s):

Remote Sensing ◽

Language Processing ◽

State Of The Art ◽

Representation Learning ◽

Learning Performance ◽

Great Success ◽

Scene Classification ◽

Linear Classifier ◽

Classification Tasks ◽

Multiple Patches

Remote sensing scene classification remains challenging due to the complexity and variety of scenes. With the development of attention-based methods, Convolutional Neural Networks (CNNs) have achieved competitive performance in remote sensing scene classification tasks. As an important method of the attention-based model, the Transformer has achieved great success in the field of natural language processing. Recently, the Transformer has been used for computer vision tasks. However, most existing methods divide the original image into multiple patches and encode the patches as the input of the Transformer, which limits the model’s ability to learn the overall features of the image. In this paper, we propose a new remote sensing scene classification method, Remote Sensing Transformer (TRS), a powerful “pure CNNs→Convolution + Transformer → pure Transformers” structure. First, we integrate self-attention into ResNet in a novel way, using our proposed Multi-Head Self-Attention layer instead of 3 × 3 spatial revolutions in the bottleneck. Then we connect multiple pure Transformer encoders to further improve the representation learning performance completely depending on attention. Finally, we use a linear classifier for classification. We train our model on four public remote sensing scene datasets: UC-Merced, AID, NWPU-RESISC45, and OPTIMAL-31. The experimental results show that TRS exceeds the state-of-the-art methods and achieves higher accuracy.

Download Full-text

A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification

Journal of Healthcare Engineering ◽

10.1155/2022/3498123 ◽

2022 ◽

Vol 2022 ◽

pp. 1-17

Author(s):

Rukhma Qasim ◽

Waqas Haider Bangyal ◽

Mohammed A. Alqarni ◽

Abdulwahab Ali Almazroi

Keyword(s):

Data Mining ◽

Social Media ◽

Transfer Learning ◽

Language Processing ◽

Text Classification ◽

Hate Speech ◽

Classification Problem ◽

Learning Approaches ◽

Fake News ◽

Targeted Marketing

Text Classification problem has been thoroughly studied in information retrieval problems and data mining tasks. It is beneficial in multiple tasks including medical diagnose health and care department, targeted marketing, entertainment industry, and group filtering processes. A recent innovation in both data mining and natural language processing gained the attention of researchers from all over the world to develop automated systems for text classification. NLP allows categorizing documents containing different texts. A huge amount of data is generated on social media sites through social media users. Three datasets have been used for experimental purposes including the COVID-19 fake news dataset, COVID-19 English tweet dataset, and extremist-non-extremist dataset which contain news blogs, posts, and tweets related to coronavirus and hate speech. Transfer learning approaches do not experiment on COVID-19 fake news and extremist-non-extremist datasets. Therefore, the proposed work applied transfer learning classification models on both these datasets to check the performance of transfer learning models. Models are trained and evaluated on the accuracy, precision, recall, and F1-score. Heat maps are also generated for every model. In the end, future directions are proposed.

Download Full-text