scholarly journals Automated Paraphrase Quality Assessment Using Language Models and Transfer Learning

Computers ◽  
2021 ◽  
Vol 10 (12) ◽  
pp. 166
Author(s):  
Bogdan Nicula ◽  
Mihai Dascalu ◽  
Natalie N. Newton ◽  
Ellen Orcutt ◽  
Danielle S. McNamara
Keyword(s):  
Quality Assessment ◽  
Transfer Learning ◽  
Network Models ◽  
Learning Task ◽  
Fine Tuning ◽  
Language Models ◽  
Neural Network Models ◽  
Writing Ability ◽  
Four Dimensions ◽  
Timely Feedback

Learning to paraphrase supports both writing ability and reading comprehension, particularly for less skilled learners. As such, educational tools that integrate automated evaluations of paraphrases can be used to provide timely feedback to enhance learner paraphrasing skills more efficiently and effectively. Paraphrase identification is a popular NLP classification task that involves establishing whether two sentences share a similar meaning. Paraphrase quality assessment is a slightly more complex task, in which pairs of sentences are evaluated in-depth across multiple dimensions. In this study, we focus on four dimensions: lexical, syntactical, semantic, and overall quality. Our study introduces and evaluates various machine learning models using handcrafted features combined with Extra Trees, Siamese neural networks using BiLSTM RNNs, and pretrained BERT-based models, together with transfer learning from a larger general paraphrase corpus, to estimate the quality of paraphrases across the four dimensions. Two datasets are considered for the tasks involving paraphrase quality: ULPC (User Language Paraphrase Corpus) containing 1998 paraphrases and a smaller dataset with 115 paraphrases based on children’s inputs. The paraphrase identification dataset used for the transfer learning task is the MSRP dataset (Microsoft Research Paraphrase Corpus) containing 5801 paraphrases. On the ULPC dataset, our BERT model improves upon the previous baseline by at least 0.1 in F1-score across the four dimensions. When using fine-tuning from ULPC for the children dataset, both the BERT and Siamese neural network models improve upon their original scores by at least 0.11 F1-score. The results of these experiments suggest that transfer learning using generic paraphrase identification datasets can be successful, while at the same time obtaining comparable results in fewer epochs.

scholarly journals AdaFilter: Adaptive Filter Fine-Tuning for Deep Transfer Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 4060-4066
Author(s):  
Yunhui Guo ◽  
Yandong Li ◽  
Liqiang Wang ◽  
Tajana Rosing
Keyword(s):  
Transfer Learning ◽  
Deep Neural Networks ◽  
Network Models ◽  
Fine Tuning ◽  
Public Image ◽  
Classification Error ◽  
Neural Network Models ◽  
Learning Technique ◽  
Task Transfer ◽  
Fine Tune

There is an increasing number of pre-trained deep neural network models. However, it is still unclear how to effectively use these models for a new task. Transfer learning, which aims to transfer knowledge from source tasks to a target task, is an effective solution to this problem. Fine-tuning is a popular transfer learning technique for deep neural networks where a few rounds of training are applied to the parameters of a pre-trained model to adapt them to a new task. Despite its popularity, in this paper we show that fine-tuning suffers from several drawbacks. We propose an adaptive fine-tuning approach, called AdaFilter, which selects only a part of the convolutional filters in the pre-trained model to optimize on a per-example basis. We use a recurrent gated network to selectively fine-tune convolutional filters based on the activations of the previous layer. We experiment with 7 public image classification datasets and the results show that AdaFilter can reduce the average classification error of the standard fine-tuning by 2.54%.

scholarly journals CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model

2021 ◽  
Vol 1 (4) ◽  
pp. 615-637
Author(s):  
Kimia Ameri ◽  
Michael Hempel ◽  
Hamid Sharif ◽  
Juan Lopez ◽  
Kalyan Perumalla
Keyword(s):  
Language Model ◽  
Network Models ◽  
Fine Tuning ◽  
Language Models ◽  
Data Sampling ◽  
Neural Network Models ◽  
Industrial Control ◽  
Industrial Control Systems ◽  
Wide Range ◽  
The Impact

We introduce CyBERT, a cybersecurity feature claims classifier based on bidirectional encoder representations from transformers and a key component in our semi-automated cybersecurity vetting for industrial control systems (ICS). To train CyBERT, we created a corpus of labeled sequences from ICS device documentation collected across a wide range of vendors and devices. This corpus provides the foundation for fine-tuning BERT’s language model, including a prediction-guided relabeling process. We propose an approach to obtain optimal hyperparameters, including the learning rate, the number of dense layers, and their configuration, to increase the accuracy of our classifier. Fine-tuning all hyperparameters of the resulting model led to an increase in classification accuracy from 76% obtained with BertForSequenceClassification’s original architecture to 94.4% obtained with CyBERT. Furthermore, we evaluated CyBERT for the impact of randomness in the initialization, training, and data-sampling phases. CyBERT demonstrated a standard deviation of ±0.6% during validation across 100 random seed values. Finally, we also compared the performance of CyBERT to other well-established language models including GPT2, ULMFiT, and ELMo, as well as neural network models such as CNN, LSTM, and BiLSTM. The results showed that CyBERT outperforms these models on the validation accuracy and the F1 score, validating CyBERT’s robustness and accuracy as a cybersecurity feature claims classifier.

scholarly journals Transfer Learning for Risk Classification of Social Media Posts: Model Evaluation Study (Preprint)

2019 ◽  
Author(s):  
Derek Howard ◽  
Marta M Maslej ◽  
Justin Lee ◽  
Jacob Ritchie ◽  
Geoffrey Woollard ◽  
...  
Keyword(s):  
Mental Health ◽  
Machine Learning ◽  
Social Media ◽  
Transfer Learning ◽  
Computational Linguistics ◽  
Feature Representation ◽  
Fine Tuning ◽  
Language Models ◽  
Universal Sentence ◽  
Text Feature

BACKGROUND Mental illness affects a significant portion of the worldwide population. Online mental health forums can provide a supportive environment for those afflicted and also generate a large amount of data that can be mined to predict mental health states using machine learning methods. OBJECTIVE This study aimed to benchmark multiple methods of text feature representation for social media posts and compare their downstream use with automated machine learning (AutoML) tools. We tested on datasets that contain posts labeled for perceived suicide risk or moderator attention in the context of self-harm. Specifically, we assessed the ability of the methods to prioritize posts that a moderator would identify for immediate response. METHODS We used 1588 labeled posts from the Computational Linguistics and Clinical Psychology (CLPsych) 2017 shared task collected from the Reachout.com forum. Posts were represented using lexicon-based tools, including Valence Aware Dictionary and sEntiment Reasoner, Empath, and Linguistic Inquiry and Word Count, and also using pretrained artificial neural network models, including DeepMoji, Universal Sentence Encoder, and Generative Pretrained Transformer-1 (GPT-1). We used Tree-based Optimization Tool and Auto-Sklearn as AutoML tools to generate classifiers to triage the posts. RESULTS The top-performing system used features derived from the GPT-1 model, which was fine-tuned on over 150,000 unlabeled posts from Reachout.com. Our top system had a macroaveraged F1 score of 0.572, providing a new state-of-the-art result on the CLPsych 2017 task. This was achieved without additional information from metadata or preceding posts. Error analyses revealed that this top system often misses expressions of hopelessness. In addition, we have presented visualizations that aid in the understanding of the learned classifiers. CONCLUSIONS In this study, we found that transfer learning is an effective strategy for predicting risk with relatively little labeled data and noted that fine-tuning of pretrained language models provides further gains when large amounts of unlabeled text are available.

scholarly journals Importance-Aware Learning for Neural Headline Editing

2020 ◽  
Vol 34 (05) ◽  
pp. 9282-9289
Author(s):  
Qingyang Wu ◽  
Lei Li ◽  
Hao Zhou ◽  
Ying Zeng ◽  
Zhou Yu
Keyword(s):  
Social Media ◽  
Large Scale ◽  
Network Models ◽  
Language Models ◽  
Neural Network Models ◽  
Generation Task ◽  
Social Media Platforms ◽  
Editing Process ◽  
Different Levels

Many social media news writers are not professionally trained. Therefore, social media platforms have to hire professional editors to adjust amateur headlines to attract more readers. We propose to automate this headline editing process through neural network models to provide more immediate writing support for these social media news writers. To train such a neural headline editing model, we collected a dataset which contains articles with original headlines and professionally edited headlines. However, it is expensive to collect a large number of professionally edited headlines. To solve this low-resource problem, we design an encoder-decoder model which leverages large scale pre-trained language models. We further improve the pre-trained model's quality by introducing a headline generation task as an intermediate task before the headline editing task. Also, we propose Self Importance-Aware (SIA) loss to address the different levels of editing in the dataset by down-weighting the importance of easily classified tokens and sentences. With the help of Pre-training, Adaptation, and SIA, the model learns to generate headlines in the professional editor's style. Experimental results show that our method significantly improves the quality of headline editing comparing against previous methods.

scholarly journals Recognition of mango leaf disease using convolutional neural network models: a transfer learning approach

2021 ◽  
Vol 23 (3) ◽  
pp. 1681
Author(s):  
Aditya Rajbongshi ◽  
Thaharim Khan ◽  
Md. Mahbubur Rahman ◽  
Anik Pramanik ◽  
Shah Md Tanvir Siddiquee ◽  
...  
Keyword(s):  
Transfer Learning ◽  
Image Data ◽  
Network Models ◽  
Plant Diseases ◽  
Prevention Measures ◽  
Neural Network Models ◽  
Data Set ◽  
Learning Techniques ◽  
Overall Performance ◽  
Mango Leaves

<p>The acknowledgment of plant diseases assumes an indispensable part in taking infectious prevention measures to improve the quality and amount of harvest yield. Mechanization of plant diseases is a lot advantageous as it decreases the checking work in an enormous cultivated area where mango is planted to a huge extend. Leaves being the food hotspot for plants, the early and precise recognition of leaf diseases is significant. This work focused on grouping and distinguishing the diseases of mango leaves through the process of CNN. DenseNet201, InceptionResNetV2, InceptionV3, ResNet50, ResNet152V2, and Xception all these models of CNN with transfer learning techniques are used here for getting better accuracy from the targeted data set. Image acquisition, image segmentation, and features extraction are the steps involved in disease detection. Different kinds of leaf diseases which are considered as the class for this work such as anthracnose, gall machi, powdery mildew, red rust are used in the dataset consisting of 1500 images of diseased and also healthy mango leaves image data another class is also added in the dataset. We have also evaluated the overall performance matrices and found that the DenseNet201 outperforms by obtaining the highest accuracy as 98.00% than other models.</p>

scholarly journals Convolutional Neural Network for Remote-Sensing Scene Classification: Transfer Learning Analysis

2019 ◽  
Vol 12 (1) ◽  
pp. 86 ◽  
Author(s):  
Rafael Pires de Lima ◽  
Kurt Marfurt
Keyword(s):  
Neural Network ◽  
Remote Sensing ◽  
Convolutional Neural Network ◽  
Transfer Learning ◽  
Network Models ◽  
Remote Sensing Image ◽  
Natural Images ◽  
Natural Image ◽  
Scene Classification ◽  
Neural Network Models

Remote-sensing image scene classification can provide significant value, ranging from forest fire monitoring to land-use and land-cover classification. Beginning with the first aerial photographs of the early 20th century to the satellite imagery of today, the amount of remote-sensing data has increased geometrically with a higher resolution. The need to analyze these modern digital data motivated research to accelerate remote-sensing image classification. Fortunately, great advances have been made by the computer vision community to classify natural images or photographs taken with an ordinary camera. Natural image datasets can range up to millions of samples and are, therefore, amenable to deep-learning techniques. Many fields of science, remote sensing included, were able to exploit the success of natural image classification by convolutional neural network models using a technique commonly called transfer learning. We provide a systematic review of transfer learning application for scene classification using different datasets and different deep-learning models. We evaluate how the specialization of convolutional neural network models affects the transfer learning process by splitting original models in different points. As expected, we find the choice of hyperparameters used to train the model has a significant influence on the final performance of the models. Curiously, we find transfer learning from models trained on larger, more generic natural images datasets outperformed transfer learning from models trained directly on smaller remotely sensed datasets. Nonetheless, results show that transfer learning provides a powerful tool for remote-sensing scene classification.

scholarly journals Tongue fissure visualization by using deep learning – an example of the application of artificial intelligence in traditional medicine

2020 ◽  
Author(s):  
Wen-Hsien Chang ◽  
Han-Kuei Wu ◽  
Lun-chien Lo ◽  
William W. L. Hsiao ◽  
Hsueh-Ting Chu ◽  
...  
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Transfer Learning ◽  
Network Model ◽  
Neural Network Model ◽  
Graphics Processing Units ◽  
Network Models ◽  
Neural Network Models ◽  
The Neural Network ◽  
Objective Evidence

Abstract Background: Traditional Chinese medicine (TCM) describes physiological and pathological changes inside and outside the human body by the application of four methods of diagnosis. One of the four methods, tongue diagnosis, is widely used by TCM physicians, since it allows direct observations that prevent discrepancies in the patient’s history and, as such, provides clinically important, objective evidence. The clinical significance of tongue features has been explored in both TCM and modern medicine. However, TCM physicians may have different interpretations of the features displayed by the same tongue, and therefore intra- and inter-observer agreements are relatively low. If an automated interpretation system could be developed, more consistent results could be obtained, and learning could also be more efficient. This study will apply a recently developed deep learning method to the classification of tongue features, and indicate the regions where the features are located.Methods: A large number of tongue photographs with labeled fissures were used. Transfer learning was conducted using the ImageNet-pretrained ResNet50 model to determine whether tongue fissures were identified on a tongue photograph. Often, the neural network model lacks interpretability, and users cannot understand how the model determines the presence of tongue fissures. Therefore, Gradient-weighted Class Activation Mapping (Grad-CAM) was also applied to directly mark the tongue features on the tongue image. Results: Only 6 epochs were trained in this study and no graphics processing units (GPUs) were used. It took less than 4 minutes for each epoch to be trained. The correct rate for the test set was approximately 70%. After the model training was completed, Grad-CAM was applied to localize tongue fissures in each image. The neural network model not only determined whether tongue fissures existed, but also allowed users to learn about the tongue fissure regions.Conclusions: This study demonstrated how to apply transfer learning using the ImageNet-pretrained ResNet50 model for the identification and localization of tongue fissures and regions. The neural network model built in this study provided interpretability and intuitiveness, (often lacking in general neural network models), and improved the feasibility for clinical application.

scholarly journals Transfer Learning Methods as a New Approach in Computer Vision Tasks with Small Datasets

2020 ◽  
Vol 45 (3) ◽  
pp. 179-193
Author(s):  
Andrzej Brodzicki ◽  
Michal Piekarski ◽  
Dariusz Kucharski ◽  
Joanna Jaworek-Korjakowska ◽  
Marek Gorgon
Keyword(s):  
Computer Vision ◽  
Transfer Learning ◽  
Network Models ◽  
Classification Problems ◽  
Quality Of Data ◽  
Neural Network Models ◽  
Learning Methods ◽  
New Approach ◽  
Thickness Prediction

AbstractDeep learning methods, used in machine vision challenges, often face the problem of the amount and quality of data. To address this issue, we investigate the transfer learning method. In this study, we briefly describe the idea and introduce two main strategies of transfer learning. We also present the widely-used neural network models, that in recent years performed best in ImageNet classification challenges. Furthermore, we shortly describe three different experiments from computer vision field, that confirm the developed algorithms ability to classify images with overall accuracy 87.2-95%. Achieved numbers are state-of-the-art results in melanoma thickness prediction, anomaly detection and Clostridium di cile cytotoxicity classification problems.

Sign in / Sign up

Export Citation Format

Share Document