scholarly journals Enhance Text-to-Text Transfer Transformer with Generated Questions for Thai Question Answering

2021 ◽  
Vol 11 (21) ◽  
pp. 10267
Author(s):  
Puri Phakmongkol ◽  
Peerapon Vateekul
Keyword(s):  
Language Processing ◽  
English Language ◽  
Question Answering ◽  
Model Performance ◽  
Fine Tuning ◽  
Exact Match ◽  
Word Level ◽  
Evaluation Measure ◽  
Multilingual Text ◽  
Natural Language Processing Task

Question Answering (QA) is a natural language processing task that enables the machine to understand a given context and answer a given question. There are several QA research trials containing high resources of the English language. However, Thai is one of the languages that have low availability of labeled corpora in QA studies. According to previous studies, while the English QA models could achieve more than 90% of F1 scores, Thai QA models could obtain only 70% in our baseline. In this study, we aim to improve the performance of Thai QA models by generating more question-answer pairs with Multilingual Text-to-Text Transfer Transformer (mT5) along with data preprocessing methods for Thai. With this method, the question-answer pairs can synthesize more than 100 thousand pairs from provided Thai Wikipedia articles. Utilizing our synthesized data, many fine-tuning strategies were investigated to achieve the highest model performance. Furthermore, we have presented that the syllable-level F1 is a more suitable evaluation measure than Exact Match (EM) and the word-level F1 for Thai QA corpora. The experiment was conducted on two Thai QA corpora: Thai Wiki QA and iApp Wiki QA. The results show that our augmented model is the winner on both datasets compared to other modern transformer models: Roberta and mT5.

scholarly journals Emerging trends: A gentle introduction to fine-tuning

2021 ◽  
Vol 27 (6) ◽  
pp. 763-778
Author(s):  
Kenneth Ward Church ◽  
Zeyu Chen ◽  
Yanjun Ma
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Question Answering ◽  
General Purpose ◽  
Fine Tuning ◽  
Language Engineering ◽  
Training Models ◽  
Emerging Trends ◽  
Foundation Model ◽  
Programming Skills

AbstractThe previous Emerging Trends article (Church et al., 2021. Natural Language Engineering27(5), 631–645.) introduced deep nets to poets. Poets is an imperfect metaphor, intended as a gesture toward inclusion. The future for deep nets will benefit by reaching out to a broad audience of potential users, including people with little or no programming skills, and little interest in training models. That paper focused on inference, the use of pre-trained models, as is, without fine-tuning. The goal of this paper is to make fine-tuning more accessible to a broader audience. Since fine-tuning is more challenging than inference, the examples in this paper will require modest programming skills, as well as access to a GPU. Fine-tuning starts with a general purpose base (foundation) model and uses a small training set of labeled data to produce a model for a specific downstream application. There are many examples of fine-tuning in natural language processing (question answering (SQuAD) and GLUE benchmark), as well as vision and speech.

scholarly journals Improving the state-of-the-art in Thai semantic similarity using distributional semantics and ontological information

PLoS ONE ◽  
2021 ◽  
Vol 16 (2) ◽  
pp. e0246751
Author(s):  
Ponrudee Netisopakul ◽  
Gerhard Wohlgenannt ◽  
Aleksei Pulich ◽  
Zar Zar Hlaing
Keyword(s):  
Semantic Similarity ◽  
Language Processing ◽  
English Language ◽  
State Of The Art ◽  
Word Sense Disambiguation ◽  
Similarity Score ◽  
The State ◽  
Word Sense ◽  
Word Level ◽  
High Fraction

Research into semantic similarity has a long history in lexical semantics, and it has applications in many natural language processing (NLP) tasks like word sense disambiguation or machine translation. The task of calculating semantic similarity is usually presented in the form of datasets which contain word pairs and a human-assigned similarity score. Algorithms are then evaluated by their ability to approximate the gold standard similarity scores. Many such datasets, with different characteristics, have been created for English language. Recently, four of those were transformed to Thai language versions, namely WordSim-353, SimLex-999, SemEval-2017-500, and R&G-65. Given those four datasets, in this work we aim to improve the previous baseline evaluations for Thai semantic similarity and solve challenges of unsegmented Asian languages (particularly the high fraction of out-of-vocabulary (OOV) dataset terms). To this end we apply and integrate different strategies to compute similarity, including traditional word-level embeddings, subword-unit embeddings, and ontological or hybrid sources like WordNet and ConceptNet. With our best model, which combines self-trained fastText subword embeddings with ConceptNet Numberbatch, we managed to raise the state-of-the-art, measured with the harmonic mean of Pearson on Spearman ρ, by a large margin from 0.356 to 0.688 for TH-WordSim-353, from 0.286 to 0.769 for TH-SemEval-500, from 0.397 to 0.717 for TH-SimLex-999, and from 0.505 to 0.901 for TWS-65.

scholarly journals Investigation of Improving The Pre-Training And Fine-Tuning of BERT Model For Biomedical Relation Extraction

2021 ◽  
Author(s):  
Peng Su ◽  
K. Vijay-Shanker
Keyword(s):  
Language Processing ◽  
Domain Knowledge ◽  
Model Performance ◽  
Relation Extraction ◽  
Biomedical Literature ◽  
Fine Tuning ◽  
Score Improvement ◽  
Model Generalization ◽  
Training Stage ◽  
Benchmark Datasets

Abstract Background: Recently, automatically extracting biomedical relations has been a significant subject in biomedical research due to the rapid growth of biomedical literature. Since the adaptation to the biomedical domain, the transformer-based BERT models have produced leading results on many biomedical natural language processing tasks. In this work, we will explore the approaches to improve the BERT model for relation extraction tasks in both the pre-training and fine-tuning stages of its applications. In the pre-training stage, we add another level of BERT adaptation on sub-domain data to bridge the gap between domain knowledge and task-specific knowledge. Also, we propose methods to incorporate the ignored knowledge in the last layer of BERT to improve its fine-tuning. Results: The experiment results demonstrate that our approaches for pre-training and fine-tuning can improve the BERT model performance. After combining the two proposed techniques, our approach outperforms the original BERT models with averaged F1 score improvement of 2.1% on relation extraction tasks. Moreover, our approach achieves state-of-the-art performance on three relation extraction benchmark datasets. Conclusions: The extra pre-training step on sub-domain data can help the BERT model generalization on specific tasks, and our proposed fine-tuning mechanism could utilize the knowledge in the last layer of BERT to boost the model performance. Furthermore, the combination of these two approaches further improves the performance of BERT model on the relation extraction tasks.

scholarly journals Question Answering Model in Thai by using Squad Thai Wikipedia dataset

2021 ◽  
Author(s):  
Wicharn Rueangkhajorn ◽  
Jonathan H. Chan
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
English Language ◽  
Question Answering ◽  
Thai Language ◽  
Model Sharing ◽  
Fine Tune ◽  
Language Question ◽  
Better Than

Nowadays, Question Answering is one of the challenge applications in the Natural language processing domain. There are plenty of English language Question Answering model distributed on the model sharing website such as Hugging face hub. Unlike Thai language, there is on a few Thai language Question Answering model distributed on the model sharing website. So, we decided to fine-tune a multilingual Question Answering model to a specify language which is Thai language. The datasets that we will use for training is a Thai Wikipedia dataset from iApp Technology. We have tried to fine-tune on two multilingual model. We also create another dataset to evaluate adaptivity of the model. The result came out to be as satisfy. Both fine-tuned models perform better than base model on evaluation score. We have published Question Answering model to Hugging face hub that will allow people to using these models for others application later.

scholarly journals Big Data Management and Analytics in Scientific Programming: A Deep Learning-Based Method for Aspect Category Classification of Question-Answering-Style Reviews

2020 ◽  
Vol 2020 ◽  
pp. 1-10 ◽  
Author(s):  
Hanqian Wu ◽  
Mumu Liu ◽  
Shangbin Zhang ◽  
Zhike Wang ◽  
Siliang Cheng
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Question Answering ◽  
Product Information ◽  
Empirical Studies ◽  
Product Reviews ◽  
Related Information ◽  
Basic Task ◽  
Word Level ◽  
Sentence Level

Online product reviews are exploring on e-commerce platforms, and mining aspect-level product information contained in those reviews has great economic benefit. The aspect category classification task is a basic task for aspect-level sentiment analysis which has become a hot research topic in the natural language processing (NLP) field during the last decades. In various e-commerce platforms, there emerge various user-generated question-answering (QA) reviews which generally contain much aspect-related information of products. Although some researchers have devoted their efforts on the aspect category classification for traditional product reviews, the existing deep learning-based approaches cannot be well applied to represent the QA-style reviews. Thus, we propose a 4-dimension (4D) textual representation model based on QA interaction-level and hyperinteraction-level by modeling with different levels of the text representation, i.e., word-level, sentence-level, QA interaction-level, and hyperinteraction-level. In our experiments, the empirical studies on datasets from three domains demonstrate that our proposals perform better than traditional sentence-level representation approaches, especially in the Digit domain.

Arabic Biomedical Community Question Answering Based on Contextualized Embeddings

2021 ◽  
Vol 17 (3) ◽  
pp. 13-29
Author(s):  
Yassine El Adlouni ◽  
Noureddine En Nahnahi ◽  
Said Ouatik El Alaoui ◽  
Mohammed Meknassi ◽  
Horacio Rodríguez ◽  
...  
Keyword(s):  
Language Processing ◽  
Question Answering ◽  
State Of The Art ◽  
Fine Tuning ◽  
Neural Models ◽  
Community Question Answering ◽  
Novel Approach ◽  
Language Representation ◽  
Textual Entailment ◽  
Scarcity Of Resources

Community question answering has become increasingly important as they are practical for seeking and sharing information. Applying deep learning models often leads to good performance, but it requires an extensive amount of annotated data, a problem exacerbated for languages suffering a scarcity of resources. Contextualized language representation models have gained success due to promising results obtained on a wide array of downstream natural language processing tasks such as text classification, textual entailment, and paraphrase identification. This paper presents a novel approach by fine-tuning contextualized embeddings for a medical domain community question answering task. The authors propose an architecture combining two neural models powered by pre-trained contextual embeddings to learn a sentence representation and thereafter fine-tuned on the task to compute a score used for both ranking and classification. The experimental results on SemEval Task 3 CQA show that the model significantly outperforms the state-of-the-art models by almost 2% for the '16 edition and 1% for the '17 edition.

scholarly journals Sentiment Analysis of Tweets in Brazilian Portuguese with Convolutional Neural Networks

2019 ◽  
Vol 7 (6) ◽  
pp. 29-41
Author(s):  
Juan Manuel Adán Coello ◽  
Armando Dalla Costa Neto
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Sentiment Analysis ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
English Language ◽  
Brazilian Portuguese ◽  
The World ◽  
Better Than ◽  
Natural Language Processing Task

Sentiment analysis of texts posted on Twitter is a natural language processing task whose importance has grown along with the increase in the number of users of the platform and the interest of organizations on the opinions of their employees, customers and users.Although Brazil is the sixth country in the world with most active users of Tweeter and Portuguese is the seventh most spoken language in the world, with 221 million speakers (200 million of them living in Brazil), the number of articles that discuss sentiment analysis approaches for Brazilian Portuguese is a small fraction of those that focus on the English language. On the other hand, few works use deep learning for this task when compared with other machine learning and lexical based methods. In this context, the work described in this article addresses the problem using Convolutional Neural Networks (CNN). The paper presents the results of an experimental evaluation that shows that a CNN with a relatively simple architecture can perform much better than a previous approach that uses ensembles of other machine learning classifiers combined with text preprocessing heuristics

scholarly journals DATLMedQA: A Data Augmentation and Transfer Learning Based Solution for Medical Question Answering

2021 ◽  
Vol 11 (23) ◽  
pp. 11251
Author(s):  
Shuohua Zhou ◽  
Yanping Zhang
Keyword(s):  
Language Processing ◽  
Prediction Accuracy ◽  
Question Answering ◽  
Data Augmentation ◽  
State Of The Art ◽  
Model Performance ◽  
The Internet ◽  
Question Generation ◽  
Topic Extraction ◽  
Questions And Answers

With the outbreak of COVID-19 that has prompted an increased focus on self-care, more and more people hope to obtain disease knowledge from the Internet. In response to this demand, medical question answering and question generation tasks have become an important part of natural language processing (NLP). However, there are limited samples of medical questions and answers, and the question generation systems cannot fully meet the needs of non-professionals for medical questions. In this research, we propose a BERT medical pretraining model, using GPT-2 for question augmentation and T5-Small for topic extraction, calculating the cosine similarity of the extracted topic and using XGBoost for prediction. With augmentation using GPT-2, the prediction accuracy of our model outperforms the state-of-the-art (SOTA) model performance. Our experiment results demonstrate the outstanding performance of our model in medical question answering and question generation tasks, and its great potential to solve other biomedical question answering challenges.

scholarly journals Question Answering Model in Thai by using Squad Thai Wikipedia dataset

2021 ◽  
Author(s):  
Wicharn Rueangkhajorn ◽  
Jonathan H. Chan
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
English Language ◽  
Question Answering ◽  
Thai Language ◽  
Model Sharing ◽  
Fine Tune ◽  
Language Question ◽  
Better Than

Nowadays, Question Answering is one of the challenge applications in the Natural language processing domain. There are plenty of English language Question Answering model distributed on the model sharing website such as Hugging face hub. Unlike Thai language, there is on a few Thai language Question Answering model distributed on the model sharing website. So, we decided to fine-tune a multilingual Question Answering model to a specify language which is Thai language. The datasets that we will use for training is a Thai Wikipedia dataset from iApp Technology. We have tried to fine-tune on two multilingual model. We also create another dataset to evaluate adaptivity of the model. The result came out to be as satisfy. Both fine-tuned models perform better than base model on evaluation score. We have published Question Answering model to Hugging face hub that will allow people to using these models for others application later.

Reports of the Workshops Held at the 2019 AAAI Conference on Artificial Intelligence

AI Magazine ◽  
2019 ◽  
Vol 40 (3) ◽  
pp. 67-78
Author(s):  
Guy Barash ◽  
Mauricio Castillo-Effen ◽  
Niyati Chhaya ◽  
Peter Clark ◽  
Huáscar Espinoza ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Language Processing ◽  
Cyber Security ◽  
Question Answering ◽  
Intent Recognition ◽  
Affective Content ◽  
Learning Plan ◽  
Dialog System ◽  
Affective Content Analysis ◽  
Games And Simulations

The workshop program of the Association for the Advancement of Artificial Intelligence’s 33rd Conference on Artificial Intelligence (AAAI-19) was held in Honolulu, Hawaii, on Sunday and Monday, January 27–28, 2019. There were fifteen workshops in the program: Affective Content Analysis: Modeling Affect-in-Action, Agile Robotics for Industrial Automation Competition, Artificial Intelligence for Cyber Security, Artificial Intelligence Safety, Dialog System Technology Challenge, Engineering Dependable and Secure Machine Learning Systems, Games and Simulations for Artificial Intelligence, Health Intelligence, Knowledge Extraction from Games, Network Interpretability for Deep Learning, Plan, Activity, and Intent Recognition, Reasoning and Learning for Human-Machine Dialogues, Reasoning for Complex Question Answering, Recommender Systems Meet Natural Language Processing, Reinforcement Learning in Games, and Reproducible AI. This report contains brief summaries of the all the workshops that were held.

Sign in / Sign up

Export Citation Format

Share Document