scholarly journals Transformer-Based Language Model Fine-Tuning Methods for COVID-19 Fake News Detection

2021 ◽  
pp. 83-92
Author(s):  
Ben Chen ◽  
Bin Chen ◽  
Dehong Gao ◽  
Qijin Chen ◽  
Chengfu Huo ◽  
...  
Keyword(s):  
Language Model ◽  
Fine Tuning ◽  
Fake News ◽  
Tuning Methods

scholarly journals Tuned bidirectional encoder representations from transformers for fake news detection

2021 ◽  
Vol 22 (3) ◽  
pp. 1667
Author(s):  
Amsal Pardamean ◽  
Hilman F. Pardede
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Traditional Method ◽  
Language Model ◽  
Learning Technologies ◽  
Fine Tuning ◽  
Fake News ◽  
Source Of Information ◽  
Better Than ◽  
Made In

Online medias are currently the dominant source of Information due to not being limited by time and place, fast and wide distributions. However, inaccurate news, or often referred as fake news is a major problem in news dissemination for online medias. Inaccurate news is information that is not true, that is engineered to cover the real information and has no factual basis. Usually, inaccurate news is made in the form of news that has mass appeal and is presented in the guise of genuine and legitimate news nuances to deceive or change the reader's mind or opinion. Identification of inaccurate news from real news can be done with natural language processing (NLP) technologies. In this paper, we proposed bidirectional encoder representations from transformers (BERT) for inaccurate news identification. BERT is a language model based on deep learning technologies and it has found effective for many NLP tasks. In this study, we use transfer learning and fine-tuning to adapt BERT for inaccurate news identification. The experiments show that our method could achieve accuracy of 99.23%, recall 99.46%, precision 98.86%, and F-Score of 99.15%. It is largely better than traditional method for the same tasks.

scholarly journals Adapting Pre-trained Language Models to Rumor Detection on Twitter

2021 ◽  
Vol 27 (10) ◽  
pp. 1128-1148
Author(s):  
Hamda Slimi ◽  
Ibrahim Bounhas ◽  
Yahya Slimani
Keyword(s):  
State Of The Art ◽  
Language Model ◽  
Fine Tuning ◽  
Language Models ◽  
Fake News ◽  
Timely Manner ◽  
Fast Pace ◽  
Social Media Platforms ◽  
Rumor Detection ◽  
Malicious Intent

Fake news has invaded social media platforms where false information is being propagated with malicious intent at a fast pace. These circumstances required the development of solutions to monitor and detect rumor in a timely manner. In this paper, we propose an approach that seeks to detect emerging and unseen rumors on Twitter by adapting a pre-trained language model to the task of rumor detection, namely RoBERTa. A comparison against content-based characteristics has shown the capability of the model to surpass handcrafted features. Experimental results show that our approach outperforms state of the art ones in all metrics and that the fine tuning of RoBERTa led to richer word embeddings that consistently and significantly enhance the precision of rumor recognition.

scholarly journals Classification of Fake News by Fine-tuning Deep Bidirectional Transformers based Language Model

2018 ◽  
pp. 163973
Author(s):  
Akshay Aggarwal ◽  
Aniruddha Chauhan ◽  
Deepika Kumar ◽  
Mamta Mittal ◽  
Sharad Verma
Keyword(s):  
Language Model ◽  
Fine Tuning ◽  
Fake News

scholarly journals Enhanced Language Model with Hybrid Knowledge Graph for Mathematical Topic Prediction

2021 ◽  
Author(s):  
Minghui Wu ◽  
Canghong Jin ◽  
Wenkang Hu ◽  
Yabo Chen
Keyword(s):  
Language Model ◽  
Real Data ◽  
Mathematical Concept ◽  
Fine Tuning ◽  
Knowledge Graph ◽  
Mathematics Knowledge ◽  
Hybrid Knowledge ◽  
Model Set ◽  
Set Up ◽  
Mathematical Topic

Understanding mathematical topics is important for both educators and students to capture latent concepts of questions, evaluate study performance, and recommend content in online learning systems. Compared to traditional text classification, mathematical topic classification has several main challenges: (1) the length of mathematical questions is relatively short; (2) there are various representations of the same mathematical concept(i.e., calculations and application); (3) the content of question is complex including algebra, geometry, and calculus. In order to overcome these problems, we propose a framework that combines content tokens and mathematical knowledge concepts in whole procedures. We embed entities from mathematics knowledge graphs, integrate entities into tokens in a masked language model, set up semantic similarity-based tasks for next-sentence prediction, and fuse knowledge vectors and token vectors during the fine-tuning procedure. We also build a Chinese mathematical topic prediction dataset consisting of more than 70,000 mathematical questions with topics. Our experiments using real data demonstrate that our knowledge graph-based mathematical topic prediction model outperforms other state-of-the-art methods.

scholarly journals "LeukNet" - A Model of Convolutional Neural Network for the Diagnosis of Leukemia

2020 ◽  
Author(s):  
Luis H. S. Vogado ◽  
Rodrigo M. S. Veras ◽  
Kelson R. T. Aires
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Cross Validation ◽  
Data Augmentation ◽  
Fine Tuning ◽  
Current State ◽  
Leukemia Diagnosis ◽  
New Guidelines ◽  
Texture Contrast ◽  
Tuning Methods

Leukemia is a disorder that affects the bone marrow, causing uncontrolled production of leukocytes, impairing the transport of oxygen and causing blood coagulation problems. In this article, we propose a new computational tool, named LeukNet, a Convolutional Neural Network (CNN) architecture based on the VGG-16 convolutional blocks, to facilitate the leukemia diagnosis from blood smear images. We evaluated different architectures and fine-tuning methods using 18 datasets containing 3536 images with distinct characteristics of color, texture, contrast, and resolution. Additionally, data augmentation operations were applied to increase the training set by up to 20 times. The k-fold cross-validation (k = 5) results achieved 98.28% of accuracy. A cross-dataset validation technique, named LeaveOne-Dataset-Out Cross-Validation (LODOCV), is also proposed to evaluate the developed model’s generalization capability. The accuracy of using LODOCV on the ALL-IDB 1, ALL-IDB 2, and UFG datasets was 97.04%, 82.46%, and 70.24%, respectively, overcoming the current state-of-the-art results and offering new guidelines for image-based computer-aided diagnosis (CAD) systems in this area.

scholarly journals Application of Smartwatches in Elderly Care with Indoor Localization Functionality

2021 ◽  
Vol 15 (05) ◽  
pp. 174
Author(s):  
Laszlo Arvai
Keyword(s):  
Mobile Technology ◽  
Indoor Localization ◽  
Elderly Care ◽  
Health Condition ◽  
General Purpose ◽  
Fine Tuning ◽  
Motion Patterns ◽  
Motion Activity ◽  
Processing Algorithms ◽  
Tuning Methods

The recent achievements in mobile technology and wearable OS makes possible to create comfortably wearable and very capable smartwatches. They have many different sensors and powerful hardware combined with general purpose OS and all this available for reasonable price. It makes it ideal device for elderly care. Monitoring the elderly’s basic health condition is very straightforward, but using smartwatch as an indoor localization device, monitoring the motion activity, recognizing the typical motion patterns of wandering is not simple. Even those watches are really capable devices, they are not equipped with direct indoor localization sensors and we would like to avoid installing special equipment’s, markers, transmitters in the home of elderly. Using only a commercially available smartwatch hardware for indoor localization is a challenging task, several filtering and data processing algorithms needs to be combined in order to provide acceptable indoor localization function. The algorithms, their connection and fine-tuning methods are explained in this article.

scholarly journals Analysis and Evaluation of Language Models for Word Sense Disambiguation

2021 ◽  
pp. 1-55
Author(s):  
Daniel Loureiro ◽  
Kiamehr Rezaee ◽  
Mohammad Taher Pilehvar ◽  
Jose Camacho-Collados
Keyword(s):  
Feature Extraction ◽  
Word Sense Disambiguation ◽  
Language Model ◽  
Training Data ◽  
Fine Tuning ◽  
Language Models ◽  
Coarse Grained ◽  
Word Sense ◽  
Sense Disambiguation ◽  
High Level

Abstract Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations in encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT can accurately capture high-level sense distinctions, even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model based WSD strategies, i.e., fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data. In fact, the simple feature extraction strategy of averaging contextualized embeddings proves robust even using only three training sentences per word sense, with minimal improvements obtained by increasing the size of this training data.

scholarly journals Selecting Informative Contexts Improves Language Model Fine-tuning

2021 ◽  
Author(s):  
Richard Antonello ◽  
Nicole Beckage ◽  
Javier Turek ◽  
Alexander Huth
Keyword(s):  
Language Model ◽  
Fine Tuning

scholarly journals Assessment of Word-Level Neural Language Models for Sentence Completion

2020 ◽  
Vol 10 (4) ◽  
pp. 1340
Author(s):  
Heewoong Park ◽  
Jonghun Park
Keyword(s):  
Language Model ◽  
Fine Tuning ◽  
Language Models ◽  
Sentence Completion ◽  
Korean Language ◽  
Learning Framework ◽  
Scholastic Aptitude ◽  
Word Level ◽  
Network Language ◽  
Comprehensive Study

The task of sentence completion, which aims to infer the missing text of a given sentence, was carried out to assess the reading comprehension level of machines as well as humans. In this work, we conducted a comprehensive study of various approaches for the sentence completion based on neural language models, which have been advanced in recent years. First, we revisited the recurrent neural network language model (RNN LM), achieving highly competitive results with an appropriate network structure and hyper-parameters. This paper presents a bidirectional version of RNN LM, which surpassed the previous best results on Microsoft Research (MSR) Sentence Completion Challenge and the Scholastic Aptitude Test (SAT) sentence completion questions. In parallel with directly applying RNN LM to sentence completion, we also employed a supervised learning framework that fine-tunes a large pre-trained transformer-based LM with a few sentence-completion examples. By fine-tuning a pre-trained BERT model, this work established state-of-the-art results on the MSR and SAT sets. Furthermore, we performed similar experimentation on newly collected cloze-style questions in the Korean language. The experimental results reveal that simply applying the multilingual BERT models for the Korean dataset was not satisfactory, which leaves room for further research.

scholarly journals Semantics-Aware BERT for Language Understanding

2020 ◽  
Vol 34 (05) ◽  
pp. 9628-9635
Author(s):  
Zhuosheng Zhang ◽  
Yuwei Wu ◽  
Hai Zhao ◽  
Zuchao Li ◽  
Shuailiang Zhang ◽  
...  
Keyword(s):  
Reading Comprehension ◽  
Natural Language ◽  
Language Model ◽  
Fine Tuning ◽  
Semantic Role Labeling ◽  
Language Understanding ◽  
Context Sensitive ◽  
Language Representation ◽  
Model Training ◽  
Machine Reading

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

Sign in / Sign up

Export Citation Format

Share Document