scholarly journals Towards Improving Open Student Answer Assessment using Pretrained Transformers

2021 ◽  
Vol 34 (1) ◽  
Author(s):  
Nisrine Ait Khayi ◽  
Vasile Rus ◽  
Lasang Tamang
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Transfer Learning ◽  
Language Processing ◽  
Text Classification ◽  
Question Answering ◽  
State Of The Art ◽  
Language Models ◽  
Assessment Task ◽  
Fine Tune

The transfer learning pretraining-finetuning  paradigm has revolutionized the natural language processing field yielding state-of the art results in  several subfields such as text classification and question answering. However, little work has been done investigating pretrained language models for the  open student answer assessment task. In this paper, we fine tune pretrained T5, BERT, RoBERTa, DistilBERT, ALBERT and XLNet models on the DT-Grade dataset which contains freely generated (or open) student answers together with judgment of their correctness. The experimental results demonstrated the effectiveness of these models based on the transfer learning pretraining-finetuning paradigm for open student answer assessment. An improvement of 8%-15% in accuracy was obtained over previous methods. Particularly, a T5 based method led to state-of-the-art results with an accuracy and F1 score of 0.88.

scholarly journals Evaluating Multilingual BERT for Estonian

2020 ◽  
Author(s):  
Claudia Kittask ◽  
Kirill Milintsevich ◽  
Kairit Sirts
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Text Classification ◽  
State Of The Art ◽  
Language Models ◽  
Neural Models ◽  
Comparable Level ◽  
Art Performance ◽  
Multiple Languages

Recently, large pre-trained language models, such as BERT, have reached state-of-the-art performance in many natural language processing tasks, but for many languages, including Estonian, BERT models are not yet available. However, there exist several multilingual BERT models that can handle multiple languages simultaneously and that have been trained also on Estonian data. In this paper, we evaluate four multilingual models—multilingual BERT, multilingual distilled BERT, XLM and XLM-RoBERTa—on several NLP tasks including POS and morphological tagging, NER and text classification. Our aim is to establish a comparison between these multilingual BERT models and the existing baseline neural models for these tasks. Our results show that multilingual BERT models can generalise well on different Estonian NLP tasks outperforming all baselines models for POS and morphological tagging and text classification, and reaching the comparable level with the best baseline for NER, with XLM-RoBERTa achieving the highest results compared with other multilingual models.

scholarly journals Transfer Learning for Classifying Spanish and English Text by Clinical Specialties

2021 ◽  
Author(s):  
Alexandra Pomares-Quimbaya ◽  
Pilar López-Úbeda ◽  
Stefan Schulz
Keyword(s):  
Natural Language Processing ◽  
Transfer Learning ◽  
Language Processing ◽  
Text Classification ◽  
State Of The Art ◽  
Domain Model ◽  
Language Models ◽  
Biomedical Domain ◽  
Spanish And English Text ◽  
General Domain

Transfer learning has demonstrated its potential in natural language processing tasks, where models have been pre-trained on large corpora and then tuned to specific tasks. We applied pre-trained transfer models to a Spanish biomedical document classification task. The main goal is to analyze the performance of text classification by clinical specialties using state-of-the-art language models for Spanish, and compared them with the results using corresponding models in English and with the most important pre-trained model for the biomedical domain. The outcomes present interesting perspectives on the performance of language models that are pre-trained for a particular domain. In particular, we found that BioBERT achieved better results on Spanish texts translated into English than the general domain model in Spanish and the state-of-the-art multilingual model.

scholarly journals Text: An R-package for Analyzing and Visualizing Human Language Using Natural Language Processing and Deep Learning

2021 ◽  
Author(s):  
Oscar Nils Erik Kjell ◽  
H. Andrew Schwartz ◽  
Salvatore Giorgi
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Rating Scale ◽  
State Of The Art ◽  
R Package ◽  
Language Models ◽  
Categorical Variables ◽  
Human Language

The language that individuals use for expressing themselves contains rich psychological information. Recent significant advances in Natural Language Processing (NLP) and Deep Learning (DL), namely transformers, have resulted in large performance gains in tasks related to understanding natural language such as machine translation. However, these state-of-the-art methods have not yet been made easily accessible for psychology researchers, nor designed to be optimal for human-level analyses. This tutorial introduces text (www.r-text.org), a new R-package for analyzing and visualizing human language using transformers, the latest techniques from NLP and DL. Text is both a modular solution for accessing state-of-the-art language models and an end-to-end solution catered for human-level analyses. Hence, text provides user-friendly functions tailored to test hypotheses in social sciences for both relatively small and large datasets. This tutorial describes useful methods for analyzing text, providing functions with reliable defaults that can be used off-the-shelf as well as providing a framework for the advanced users to build on for novel techniques and analysis pipelines. The reader learns about six methods: 1) textEmbed: to transform text to traditional or modern transformer-based word embeddings (i.e., numeric representations of words); 2) textTrain: to examine the relationships between text and numeric/categorical variables; 3) textSimilarity and 4) textSimilarityTest: to computing semantic similarity scores between texts and significance test the difference in meaning between two sets of texts; and 5) textProjection and 6) textProjectionPlot: to examine and visualize text within the embedding space according to latent or specified construct dimensions (e.g., low to high rating scale scores).

scholarly journals Inducing Relational Knowledge from BERT

2020 ◽  
Vol 34 (05) ◽  
pp. 7456-7463 ◽  
Author(s):  
Zied Bouraoui ◽  
Jose Camacho-Collados ◽  
Steven Schockaert
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Language Model ◽  
Language Models ◽  
Word Embeddings ◽  
Relational Knowledge ◽  
Wide Range ◽  
Fine Tune ◽  
Standard Word

One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.

scholarly journals A Comprehensive Exploration of Pre-training Language Models

2021 ◽  
Author(s):  
Tong Guo
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Contextual Information ◽  
Experimental Results ◽  
Language Models

Recently, the development of pre-trained language models has brought natural language processing (NLP) tasks to the new state-of-the-art. In this paper we explore the efficiency of various pre-trained language models. We pre-train a list of transformer-based models with the same amount of text and the same training steps. The experimental results shows that the most improvement upon the origin BERT is adding the RNN-layer to capture more contextual information for the transformer-encoder layers.

scholarly journals Question Answering Model in Thai by using Squad Thai Wikipedia dataset

2021 ◽  
Author(s):  
Wicharn Rueangkhajorn ◽  
Jonathan H. Chan
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
English Language ◽  
Question Answering ◽  
Thai Language ◽  
Model Sharing ◽  
Fine Tune ◽  
Language Question ◽  
Better Than

Nowadays, Question Answering is one of the challenge applications in the Natural language processing domain. There are plenty of English language Question Answering model distributed on the model sharing website such as Hugging face hub. Unlike Thai language, there is on a few Thai language Question Answering model distributed on the model sharing website. So, we decided to fine-tune a multilingual Question Answering model to a specify language which is Thai language. The datasets that we will use for training is a Thai Wikipedia dataset from iApp Technology. We have tried to fine-tune on two multilingual model. We also create another dataset to evaluate adaptivity of the model. The result came out to be as satisfy. Both fine-tuned models perform better than base model on evaluation score. We have published Question Answering model to Hugging face hub that will allow people to using these models for others application later.

scholarly journals A Comprehensive Exploration of Pre-training Language Models

2021 ◽  
Author(s):  
Tong Guo
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Contextual Information ◽  
Experimental Results ◽  
Language Models

Recently, the development of pre-trained language models has brought natural language processing (NLP) tasks to the new state-of-the-art. In this paper we explore the efficiency of various pre-trained language models. We pre-train a list of transformer-based models with the same amount of text and the same training steps. The experimental results shows that the most improvement upon the origin BERT is adding the RNN-layer to capture more contextual information for the transformer-encoder layers.

scholarly journals Deep Learning-Based Natural Language Processing for Screening Psychiatric Patients

2021 ◽  
Vol 11 ◽  
Author(s):  
Hong-Jie Dai ◽  
Chu-Hsien Su ◽  
You-Qian Lee ◽  
You-Chen Zhang ◽  
Chen-Kai Wang ◽  
...  
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Transfer Learning ◽  
Language Processing ◽  
Medical Center ◽  
Psychiatric Patients ◽  
Language Models ◽  
Great Opportunity ◽  
Problem Transformation

The introduction of pre-trained language models in natural language processing (NLP) based on deep learning and the availability of electronic health records (EHRs) presents a great opportunity to transfer the “knowledge” learned from data in the general domain to enable the analysis of unstructured textual data in clinical domains. This study explored the feasibility of applying NLP to a small EHR dataset to investigate the power of transfer learning to facilitate the process of patient screening in psychiatry. A total of 500 patients were randomly selected from a medical center database. Three annotators with clinical experience reviewed the notes to make diagnoses for major/minor depression, bipolar disorder, schizophrenia, and dementia to form a small and highly imbalanced corpus. Several state-of-the-art NLP methods based on deep learning along with pre-trained models based on shallow or deep transfer learning were adapted to develop models to classify the aforementioned diseases. We hypothesized that the models that rely on transferred knowledge would be expected to outperform the models learned from scratch. The experimental results demonstrated that the models with the pre-trained techniques outperformed the models without transferred knowledge by micro-avg. and macro-avg. F-scores of 0.11 and 0.28, respectively. Our results also suggested that the use of the feature dependency strategy to build multi-labeling models instead of problem transformation is superior considering its higher performance and simplicity in the training process.

scholarly journals Towards the Natural Language Processing as Spelling Correction for Offline Handwritten Text Recognition Systems

2020 ◽  
Vol 10 (21) ◽  
pp. 7711
Author(s):  
Arthur Flor de Sousa Neto ◽  
Byron Leite Dantas Bezerra ◽  
Alejandro Héctor Toselli
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Network Architecture ◽  
State Of The Art ◽  
Language Models ◽  
Text Recognition ◽  
Spelling Correction ◽  
Handwritten Text ◽  
Handwritten Text Recognition

The increasing portability of physical manuscripts to the digital environment makes it common for systems to offer automatic mechanisms for offline Handwritten Text Recognition (HTR). However, several scenarios and writing variations bring challenges in recognition accuracy, and, to minimize this problem, optical models can be used with language models to assist in decoding text. Thus, with the aim of improving results, dictionaries of characters and words are generated from the dataset and linguistic restrictions are created in the recognition process. In this way, this work proposes the use of spelling correction techniques for text post-processing to achieve better results and eliminate the linguistic dependence between the optical model and the decoding stage. In addition, an encoder–decoder neural network architecture in conjunction with a training methodology are developed and presented to achieve the goal of spelling correction. To demonstrate the effectiveness of this new approach, we conducted an experiment on five datasets of text lines, widely known in the field of HTR, three state-of-the-art Optical Models for text recognition and eight spelling correction techniques, among traditional statistics and current approaches of neural networks in the field of Natural Language Processing (NLP). Finally, our proposed spelling correction model is analyzed statistically through HTR system metrics, reaching an average sentence correction of 54% higher than the state-of-the-art method of decoding in the tested datasets.

scholarly journals Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

2022 ◽  
Vol 3 (1) ◽  
pp. 1-23
Author(s):  
Yu Gu ◽  
Robert Tinn ◽  
Hao Cheng ◽  
Michael Lucas ◽  
Naoto Usuyama ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Fine Tuning ◽  
Entity Recognition ◽  
Language Models ◽  
General Domain ◽  
Domain Specific ◽  
And Task

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .

Sign in / Sign up

Export Citation Format

Share Document