DATLMedQA: A Data Augmentation and Transfer Learning Based Solution for Medical Question Answering

With the outbreak of COVID-19 that has prompted an increased focus on self-care, more and more people hope to obtain disease knowledge from the Internet. In response to this demand, medical question answering and question generation tasks have become an important part of natural language processing (NLP). However, there are limited samples of medical questions and answers, and the question generation systems cannot fully meet the needs of non-professionals for medical questions. In this research, we propose a BERT medical pretraining model, using GPT-2 for question augmentation and T5-Small for topic extraction, calculating the cosine similarity of the extracted topic and using XGBoost for prediction. With augmentation using GPT-2, the prediction accuracy of our model outperforms the state-of-the-art (SOTA) model performance. Our experiment results demonstrate the outstanding performance of our model in medical question answering and question generation tasks, and its great potential to solve other biomedical question answering challenges.

Download Full-text

A Review on Question Generation from Natural Language Text

ACM Transactions on Information Systems ◽

10.1145/3468889 ◽

2022 ◽

Vol 40 (1) ◽

pp. 1-43

Author(s):

Ruqing Zhang ◽

Jiafeng Guo ◽

Lu Chen ◽

Yixing Fan ◽

Xueqi Cheng

Keyword(s):

Natural Language ◽

Question Answering ◽

Data Augmentation ◽

Text Structure ◽

Current Status ◽

Question Generation ◽

Natural Language Text ◽

Question Answering Systems ◽

The Right ◽

Language Text

Question generation is an important yet challenging problem in Artificial Intelligence (AI), which aims to generate natural and relevant questions from various input formats, e.g., natural language text, structure database, knowledge base, and image. In this article, we focus on question generation from natural language text, which has received tremendous interest in recent years due to the widespread applications such as data augmentation for question answering systems. During the past decades, many different question generation models have been proposed, from traditional rule-based methods to advanced neural network-based methods. Since there have been a large variety of research works proposed, we believe it is the right time to summarize the current status, learn from existing methodologies, and gain some insights for future development. In contrast to existing reviews, in this survey, we try to provide a more comprehensive taxonomy of question generation tasks from three different perspectives, i.e., the types of the input context text, the target answer, and the generated question. We take a deep look into existing models from different dimensions to analyze their underlying ideas, major design principles, and training strategies We compare these models through benchmark tasks to obtain an empirical understanding of the existing techniques. Moreover, we discuss what is missing in the current literature and what are the promising and desired future directions.

Download Full-text

Self-Supervised Contextual Data Augmentation for Natural Language Processing

Symmetry ◽

10.3390/sym11111393 ◽

2019 ◽

Vol 11 (11) ◽

pp. 1393

Author(s):

Dongju Park ◽

Chang Wook Ahn

Keyword(s):

Supervised Learning ◽

Language Processing ◽

Recurrent Neural Networks ◽

Question Answering ◽

Data Augmentation ◽

Language Model ◽

Contextual Data ◽

External Data ◽

Label Information ◽

Benchmark Datasets

In this paper, we propose a novel data augmentation method with respect to the target context of the data via self-supervised learning. Instead of looking for the exact synonyms of masked words, the proposed method finds words that can replace the original words considering the context. For self-supervised learning, we can employ the masked language model (MLM), which masks a specific word within a sentence and obtains the original word. The MLM learns the context of a sentence through asymmetrical inputs and outputs. However, without using the existing MLM, we propose a label-masked language model (LMLM) that can include label information for the mask tokens used in the MLM to effectively use the MLM in data with label information. The augmentation method performs self-supervised learning using LMLM and then implements data augmentation through the trained model. We demonstrate that our proposed method improves the classification accuracy of recurrent neural networks and convolutional neural network-based classifiers through several experiments for text classification benchmark datasets, including the Stanford Sentiment Treebank-5 (SST5), the Stanford Sentiment Treebank-2 (SST2), the subjectivity (Subj), the Multi-Perspective Question Answering (MPQA), the Movie Reviews (MR), and the Text Retrieval Conference (TREC) datasets. In addition, since the proposed method does not use external data, it can eliminate the time spent collecting external data, or pre-training using external data.

Download Full-text

Towards relation extraction from Arabic text: a review

International Robotics & Automation Journal ◽

10.15406/iratj.2019.05.00195 ◽

2019 ◽

Vol 5 (5) ◽

pp. 212-215

Author(s):

Abeer AlArfaj

Keyword(s):

Language Processing ◽

Question Answering ◽

State Of The Art ◽

Relation Extraction ◽

Arabic Language ◽

Extraction Methods ◽

The State ◽

Semantic Relations ◽

Arabic Text ◽

Taxonomic Relation

Semantic relation extraction is an important component of ontologies that can support many applications e.g. text mining, question answering, and information extraction. However, extracting semantic relations between concepts is not trivial and one of the main challenges in Natural Language Processing (NLP) Field. The Arabic language has complex morphological, grammatical, and semantic aspects since it is a highly inflectional and derivational language, which makes task even more challenging. In this paper, we present a review of the state of the art for relation extraction from texts, addressing the progress and difficulties in this field. We discuss several aspects related to this task, considering the taxonomic and non-taxonomic relation extraction methods. Majority of relation extraction approaches implement a combination of statistical and linguistic techniques to extract semantic relations from text. We also give special attention to the state of the work on relation extraction from Arabic texts, which need further progress.

Download Full-text

Supervised attention for answer selection in community question answering

IAES International Journal of Artificial Intelligence (IJ-AI) ◽

10.11591/ijai.v9.i2.pp203-211 ◽

2020 ◽

Vol 9 (2) ◽

pp. 203

Author(s):

Thanh Thi Ha ◽

Atsuhiro Takasu ◽

Thanh Chinh Nguyen ◽

Kiem Hieu Nguyen ◽

Van Nha Nguyen ◽

...

Keyword(s):

Language Processing ◽

Question Answering ◽

Irrelevant Information ◽

Social Question ◽

Community Question Answering ◽

Basic Model ◽

Proposed Model ◽

Questions And Answers ◽

Word Attention ◽

Better Than

Answer selection is an important task in Community Question Answering (CQA). In recent years, attention-based neural networks have been extensively studied in various natural language processing problems, including question answering. This paper explores matchLSTM for answer selection in CQA. A lexical gap in CQA is more challenging as questions and answers typical contain multiple sentences, irrelevant information, and noisy expressions. In our investigation, word-by-word attention in the original model does not work well on social question-answer pairs. We propose integrating supervised attention into matchLSTM. Specifically, we leverage lexical-semantic from external to guide the learning of attention weights for question-answer pairs. The proposed model learns more meaningful attention that allows performing better than the basic model. Our performance is among the top on SemEval datasets.

Download Full-text

Capturing Greater Context for Question Generation

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i05.6440 ◽

2020 ◽

Vol 34 (05) ◽

pp. 9065-9072

Author(s):

Luu Anh Tuan ◽

Darsh Shah ◽

Regina Barzilay

Keyword(s):

Reading Comprehension ◽

Question Answering ◽

State Of The Art ◽

Attention Mechanism ◽

Dialogue Systems ◽

Question Generation ◽

Multi Stage ◽

Art Methods ◽

Relevant Context ◽

Automatic Question Generation

Automatic question generation can benefit many applications ranging from dialogue systems to reading comprehension. While questions are often asked with respect to long documents, there are many challenges with modeling such long documents. Many existing techniques generate questions by effectively looking at one sentence at a time, leading to questions that are easy and not reflective of the human process of question generation. Our goal is to incorporate interactions across multiple sentences to generate realistic questions for long documents. In order to link a broad document context to the target answer, we represent the relevant context via a multi-stage attention mechanism, which forms the foundation of a sequence to sequence model. We outperform state-of-the-art methods on question generation on three question-answering datasets - SQuAD, MS MARCO and NewsQA. 1

Download Full-text

Dependency parsing of Polish

Poznan Studies in Contemporary Linguistics ◽

10.1515/psicl-2019-0012 ◽

2019 ◽

Vol 55 (2) ◽

pp. 305-337 ◽

Cited By ~ 1

Author(s):

Alina Wróblewska ◽

Piotr Rybak

Keyword(s):

Language Processing ◽

Argument Structure ◽

Question Answering ◽

State Of The Art ◽

Dependency Parsing ◽

Dependency Theory ◽

Crucial Issue ◽

Experimental Part ◽

Predicate Argument Structure

Abstract The predicate-argument structure transparently encoded in dependency-based syntactic representations supports machine translation, question answering, information extraction, etc. The quality of dependency parsing is therefore a crucial issue in natural language processing. In the current paper we discuss the fundamental ideas of the dependency theory and provide an overview of selected dependency-based resources for Polish. Furthermore, we present some state-of-the-art dependency parsing systems whose models can be estimated on correctly annotated data. In the experimental part, we provide an in-depth evaluation of these systems on Polish data. Our results show that graph-based parsers, even those without any neural component, are better suited for Polish than transition-based parsing systems.

Download Full-text

Vision Transformers for Remote Sensing Image Classification

Remote Sensing ◽

10.3390/rs13030516 ◽

2021 ◽

Vol 13 (3) ◽

pp. 516

Author(s):

Yakoub Bazi ◽

Laila Bashmal ◽

Mohamad M. Al Rahhal ◽

Reham Al Dayil ◽

Naif Al Ajlan

Keyword(s):

Remote Sensing ◽

Language Processing ◽

Additional Data ◽

Data Augmentation ◽

State Of The Art ◽

Remote Sensing Image ◽

Classification Performance ◽

Scene Classification ◽

Remote Sensing Image Classification ◽

Augmentation Strategies

In this paper, we propose a remote-sensing scene-classification method based on vision transformers. These types of networks, which are now recognized as state-of-the-art models in natural language processing, do not rely on convolution layers as in standard convolutional neural networks (CNNs). Instead, they use multihead attention mechanisms as the main building block to derive long-range contextual relation between pixels in images. In a first step, the images under analysis are divided into patches, then converted to sequence by flattening and embedding. To keep information about the position, embedding position is added to these patches. Then, the resulting sequence is fed to several multihead attention layers for generating the final representation. At the classification stage, the first token sequence is fed to a softmax classification layer. To boost the classification performance, we explore several data augmentation strategies to generate additional data for training. Moreover, we show experimentally that we can compress the network by pruning half of the layers while keeping competing classification accuracies. Experimental results conducted on different remote-sensing image datasets demonstrate the promising capability of the model compared to state-of-the-art methods. Specifically, Vision Transformer obtains an average classification accuracy of 98.49%, 95.86%, 95.56% and 93.83% on Merced, AID, Optimal31 and NWPU datasets, respectively. While the compressed version obtained by removing half of the multihead attention layers yields 97.90%, 94.27%, 95.30% and 93.05%, respectively.

Download Full-text

MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

Transactions of the Association for Computational Linguistics ◽

10.1162/tacl_a_00433 ◽

2021 ◽

Vol 9 ◽

pp. 1389-1406

Author(s):

Shayne Longpre ◽

Yi Lu ◽

Joachim Daiber

Keyword(s):

Question Answering ◽

State Of The Art ◽

Linguistically Diverse ◽

Data Representation ◽

Independent Data ◽

Open Domain ◽

Low Resource ◽

Art Methods ◽

Questions And Answers ◽

Cross Lingual

Abstract Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open- domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on heavily curated, language- independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state- of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages.1

Download Full-text

Goal-Driven Visual Question Generation from Radiology Images

10.20944/preprints202107.0385.v1 ◽

2021 ◽

Author(s):

Mourad Sarrouti ◽

Asma Ben Abacha ◽

Dina Demner-Fushman

Keyword(s):

Natural Language ◽

Language Processing ◽

Domain Knowledge ◽

Data Augmentation ◽

Question Generation ◽

Human Evaluation ◽

Question Category ◽

Medical Domain ◽

The Impact

Visual Question Generation (VQG) from images is a rising research topic in both fields of natural language processing and computer vision. Although there are some recent efforts towards generating questions from images in the open domain, the VQG task in the medical domain has not been well-studied so far due to the lack of labeled data. In this paper, we introduce a goal-driven VQG approach for radiology images called VQGRaD that generates questions targeting specific image aspects such as modality and abnormality. In particular, we study generating natural language questions based on the visual content of the image and on additional information such as the image caption and the question category. VQGRaD encodes the dense vectors of different inputs into two latent spaces, which allows generating, for a specific question category, relevant questions about the images, with or without their captions. We also explore the impact of domain knowledge incorporation (e.g., medical entities and semantic types) and data augmentation techniques on visual question generation in the medical domain. Experiments performed on the VQA-RAD dataset of clinical visual questions showed that VQGRaD achieves 61.86% BLEU score and outperforms strong baselines. We also performed a blinded human evaluation of the grammaticality, fluency, and relevance of the generated questions. The human evaluation demonstrated the better quality of VQGRaD outputs and showed that incorporating medical entities improves the quality of the generated questions. Using the test data and evaluation process of the ImageCLEF 2020 VQA-Med challenge, we found that relying on the proposed data augmentation technique to generate new training samples by applying different kinds of transformations, can mitigate the lack of data, avoid overfitting, and bring a substantial improvement in medical VQG.

Download Full-text

Enhance Text-to-Text Transfer Transformer with Generated Questions for Thai Question Answering

Applied Sciences ◽

10.3390/app112110267 ◽

2021 ◽

Vol 11 (21) ◽

pp. 10267

Author(s):

Puri Phakmongkol ◽

Peerapon Vateekul

Keyword(s):

Language Processing ◽

English Language ◽

Question Answering ◽

Model Performance ◽

Fine Tuning ◽

Exact Match ◽

Word Level ◽

Evaluation Measure ◽

Multilingual Text ◽

Natural Language Processing Task

Question Answering (QA) is a natural language processing task that enables the machine to understand a given context and answer a given question. There are several QA research trials containing high resources of the English language. However, Thai is one of the languages that have low availability of labeled corpora in QA studies. According to previous studies, while the English QA models could achieve more than 90% of F1 scores, Thai QA models could obtain only 70% in our baseline. In this study, we aim to improve the performance of Thai QA models by generating more question-answer pairs with Multilingual Text-to-Text Transfer Transformer (mT5) along with data preprocessing methods for Thai. With this method, the question-answer pairs can synthesize more than 100 thousand pairs from provided Thai Wikipedia articles. Utilizing our synthesized data, many fine-tuning strategies were investigated to achieve the highest model performance. Furthermore, we have presented that the syllable-level F1 is a more suitable evaluation measure than Exact Match (EM) and the word-level F1 for Thai QA corpora. The experiment was conducted on two Thai QA corpora: Thai Wiki QA and iApp Wiki QA. The results show that our augmented model is the winner on both datasets compared to other modern transformer models: Roberta and mT5.

Download Full-text