scholarly journals Self-Supervised Contextual Data Augmentation for Natural Language Processing

Symmetry ◽  
2019 ◽  
Vol 11 (11) ◽  
pp. 1393
Author(s):  
Dongju Park ◽  
Chang Wook Ahn
Keyword(s):  
Supervised Learning ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Question Answering ◽  
Data Augmentation ◽  
Language Model ◽  
Contextual Data ◽  
External Data ◽  
Label Information ◽  
Benchmark Datasets

In this paper, we propose a novel data augmentation method with respect to the target context of the data via self-supervised learning. Instead of looking for the exact synonyms of masked words, the proposed method finds words that can replace the original words considering the context. For self-supervised learning, we can employ the masked language model (MLM), which masks a specific word within a sentence and obtains the original word. The MLM learns the context of a sentence through asymmetrical inputs and outputs. However, without using the existing MLM, we propose a label-masked language model (LMLM) that can include label information for the mask tokens used in the MLM to effectively use the MLM in data with label information. The augmentation method performs self-supervised learning using LMLM and then implements data augmentation through the trained model. We demonstrate that our proposed method improves the classification accuracy of recurrent neural networks and convolutional neural network-based classifiers through several experiments for text classification benchmark datasets, including the Stanford Sentiment Treebank-5 (SST5), the Stanford Sentiment Treebank-2 (SST2), the subjectivity (Subj), the Multi-Perspective Question Answering (MPQA), the Movie Reviews (MR), and the Text Retrieval Conference (TREC) datasets. In addition, since the proposed method does not use external data, it can eliminate the time spent collecting external data, or pre-training using external data.

Astrid

2020 ◽  
Vol 14 (4) ◽  
pp. 471-484
Author(s):  
Suraj Shetiya ◽  
Saravanan Thirumuruganathan ◽  
Nick Koudas ◽  
Gautam Das
Keyword(s):  
Deep Learning ◽  
Objective Function ◽  
Pattern Matching ◽  
Language Processing ◽  
Language Model ◽  
Language Models ◽  
Selectivity Estimation ◽  
Statistical Correlations ◽  
Benchmark Datasets ◽  
Traditional Approaches

Accurate selectivity estimation for string predicates is a long-standing research challenge in databases. Supporting pattern matching on strings (such as prefix, substring, and suffix) makes this problem much more challenging, thereby necessitating a dedicated study. Traditional approaches often build pruned summary data structures such as tries followed by selectivity estimation using statistical correlations. However, this produces insufficiently accurate cardinality estimates resulting in the selection of sub-optimal plans by the query optimizer. Recently proposed deep learning based approaches leverage techniques from natural language processing such as embeddings to encode the strings and use it to train a model. While this is an improvement over traditional approaches, there is a large scope for improvement. We propose Astrid, a framework for string selectivity estimation that synthesizes ideas from traditional and deep learning based approaches. We make two complementary contributions. First, we propose an embedding algorithm that is query-type (prefix, substring, and suffix) and selectivity aware. Consider three strings 'ab', 'abc' and 'abd' whose prefix frequencies are 1000, 800 and 100 respectively. Our approach would ensure that the embedding for 'ab' is closer to 'abc' than 'abd'. Second, we describe how neural language models could be used for selectivity estimation. While they work well for prefix queries, their performance for substring queries is sub-optimal. We modify the objective function of the neural language model so that it could be used for estimating selectivities of pattern matching queries. We also propose a novel and efficient algorithm for optimizing the new objective function. We conduct extensive experiments over benchmark datasets and show that our proposed approaches achieve state-of-the-art results.

LIS4: Lesk Inspired Sense Specific Semantic Similarity using WordNet

2021 ◽  
pp. 2150006
Author(s):  
Saravanakumar Kandasamy ◽  
Aswani Kumar Cherukuri
Keyword(s):  
Information Retrieval ◽  
Natural Language Processing ◽  
Natural Language ◽  
Semantic Similarity ◽  
Language Processing ◽  
Gold Standard ◽  
Question Answering ◽  
Knowledge Based ◽  
Benchmark Datasets ◽  
Processing Information

Semantic similarity quantification between concepts is one of the inevitable parts in domains like Natural Language Processing, Information Retrieval, Question Answering, etc. to understand the text and their relationships better. Last few decades, many measures have been proposed by incorporating various corpus-based and knowledge-based resources. WordNet and Wikipedia are two of the Knowledge-based resources. The contribution of WordNet in the above said domain is enormous due to its richness in defining a word and all of its relationship with others. In this paper, we proposed an approach to quantify the similarity between concepts that exploits the synsets and the gloss definitions of different concepts using WordNet. Our method considers the gloss definitions, contextual words that are helping in defining a word, synsets of contextual word and the confidence of occurrence of a word in other word’s definition for calculating the similarity. The evaluation based on different gold standard benchmark datasets shows the efficiency of our system in comparison with other existing taxonomical and definitional measures.

scholarly journals An Empirical Study of Content Understanding in Conversational Question Answering

2020 ◽  
Vol 34 (05) ◽  
pp. 7578-7585
Author(s):  
Ting-Rui Chiang ◽  
Hao-Tong Ye ◽  
Yun-Nung Chen
Keyword(s):  
Natural Language Processing ◽  
Empirical Study ◽  
Language Processing ◽  
Question Answering ◽  
Source Code ◽  
Content Understanding ◽  
Question Answering Systems ◽  
Benchmark Datasets ◽  
Context Free ◽  
Answering Questions

With a lot of work about context-free question answering systems, there is an emerging trend of conversational question answering models in the natural language processing field. Thanks to the recently collected datasets, including QuAC and CoQA, there has been more work on conversational question answering, and recent work has achieved competitive performance on both datasets. However, to best of our knowledge, two important questions for conversational comprehension research have not been well studied: 1) How well can the benchmark dataset reflect models' content understanding? 2) Do the models well utilize the conversation content when answering questions? To investigate these questions, we design different training settings, testing settings, as well as an attack to verify the models' capability of content understanding on QuAC and CoQA. The experimental results indicate some potential hazards in the benchmark datasets, QuAC and CoQA, for conversational comprehension research. Our analysis also sheds light on both what models may learn and how datasets may bias the models. With deep investigation of the task, it is believed that this work can benefit the future progress of conversation comprehension. The source code is available at https://github.com/MiuLab/CQA-Study.

scholarly journals Learning molecular dynamics with simple language model built upon long short-term memory neural network

2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Sun-Ting Tsai ◽  
En-Jui Kuo ◽  
Pratyush Tiwary
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Language Model ◽  
Short Term ◽  
Term Memory ◽  
Molecular Systems ◽  
Simple Language ◽  
Long Short Term Memory

Abstract Recurrent neural networks have led to breakthroughs in natural language processing and speech recognition. Here we show that recurrent networks, specifically long short-term memory networks can also capture the temporal evolution of chemical/biophysical trajectories. Our character-level language model learns a probabilistic model of 1-dimensional stochastic trajectories generated from higher-dimensional dynamics. The model captures Boltzmann statistics and also reproduces kinetics across a spectrum of timescales. We demonstrate how training the long short-term memory network is equivalent to learning a path entropy, and that its embedding layer, instead of representing contextual meaning of characters, here exhibits a nontrivial connectivity between different metastable states in the underlying physical system. We demonstrate our model’s reliability through different benchmark systems and a force spectroscopy trajectory for multi-state riboswitch. We anticipate that our work represents a stepping stone in the understanding and use of recurrent neural networks for understanding the dynamics of complex stochastic molecular systems.

scholarly journals A Deeper Look at Sheet Music Composer Classification Using Self-Supervised Pretraining

2021 ◽  
Vol 11 (4) ◽  
pp. 1387
Author(s):  
Daniel Yang ◽  
Kevin Ji ◽  
TJ Tsai
Keyword(s):  
Language Processing ◽  
Data Augmentation ◽  
Language Model ◽  
Classification Performance ◽  
Large Set ◽  
Sheet Music ◽  
Modern Language ◽  
Music Score ◽  
Augmentation Strategies ◽  
Small Set

This article studies a composer style classification task based on raw sheet music images. While previous works on composer recognition have relied exclusively on supervised learning, we explore the use of self-supervised pretraining methods that have been recently developed for natural language processing. We first convert sheet music images to sequences of musical words, train a language model on a large set of unlabeled musical “sentences”, initialize a classifier with the pretrained language model weights, and then finetune the classifier on a small set of labeled data. We conduct extensive experiments on International Music Score Library Project (IMSLP) piano data using a range of modern language model architectures. We show that pretraining substantially improves classification performance and that Transformer-based architectures perform best. We also introduce two data augmentation strategies and present evidence that the model learns generalizable and semantically meaningful information.

scholarly journals External features enriched model for biomedical question answering

2021 ◽  
Vol 22 (1) ◽  
Author(s):  
Gezheng Xu ◽  
Wenge Rong ◽  
Yanmeng Wang ◽  
Yuanxin Ouyang ◽  
Zhang Xiong
Keyword(s):  
Language Processing ◽  
Large Scale ◽  
Question Answering ◽  
Language Model ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Small Scale ◽  
Original Text ◽  
Specific Domain ◽  
Part Of Speech

Abstract Background Biomedical question answering (QA) is a sub-task of natural language processing in a specific domain, which aims to answer a question in the biomedical field based on one or more related passages and can provide people with accurate healthcare-related information. Recently, a lot of approaches based on the neural network and large scale pre-trained language model have largely improved its performance. However, considering the lexical characteristics of biomedical corpus and its small scale dataset, there is still much improvement room for biomedical QA tasks. Results Inspired by the importance of syntactic and lexical features in the biomedical corpus, we proposed a new framework to extract external features, such as part-of-speech and named-entity recognition, and fused them with the original text representation encoded by pre-trained language model, to enhance the biomedical question answering performance. Our model achieves an overall improvement of all three metrics on BioASQ 6b, 7b, and 8b factoid question answering tasks. Conclusions The experiments on BioASQ question answering dataset demonstrated the effectiveness of our external feature-enriched framework. It is proven by the experiments conducted that external lexical and syntactic features can improve Pre-trained Language Model’s performance in biomedical domain question answering task.

scholarly journals Semantic and Generalized Entropy Loss Functions for Semi-Supervised Deep Learning

Entropy ◽  
2020 ◽  
Vol 22 (3) ◽  
pp. 334
Author(s):  
Krzysztof Gajowniczek ◽  
Yitao Liang ◽  
Tal Friedman ◽  
Tomasz Ząbkowski ◽  
Guy Van den Broeck
Keyword(s):  
Neural Network ◽  
Supervised Learning ◽  
Loss Function ◽  
Practical Importance ◽  
Loss Functions ◽  
Generalized Entropy ◽  
Entropy Loss ◽  
Additional Information ◽  
Label Information ◽  
Benchmark Datasets

The increasing size of modern datasets combined with the difficulty of obtaining real label information (e.g., class) has made semi-supervised learning a problem of considerable practical importance in modern data analysis. Semi-supervised learning is supervised learning with additional information on the distribution of the examples or, simultaneously, an extension of unsupervised learning guided by some constraints. In this article we present a methodology that bridges between artificial neural network output vectors and logical constraints. In order to do this, we present a semantic loss function and a generalized entropy loss function (Rényi entropy) that capture how close the neural network is to satisfying the constraints on its output. Our methods are intended to be generally applicable and compatible with any feedforward neural network. Therefore, the semantic loss and generalized entropy loss are simply a regularization term that can be directly plugged into an existing loss function. We evaluate our methodology over an artificially simulated dataset and two commonly used benchmark datasets which are MNIST and Fashion-MNIST to assess the relation between the analyzed loss functions and the influence of the various input and tuning parameters on the classification accuracy. The experimental evaluation shows that both losses effectively guide the learner to achieve (near-) state-of-the-art results on semi-supervised multiclass classification.

scholarly journals Research on Automatic Question Answering of Generative Knowledge Graph Based on Pointer Network

Information ◽  
2021 ◽  
Vol 12 (3) ◽  
pp. 136
Author(s):  
Shuang Liu ◽  
Nannan Tan ◽  
Yaqian Ge ◽  
Niko Lukač
Keyword(s):  
Knowledge Base ◽  
Word Frequency ◽  
Language Processing ◽  
Question Answering ◽  
Language Model ◽  
Word List ◽  
Superior Performance ◽  
Knowledge Graph ◽  
Semantic Features ◽  
Question Answering Systems

Question-answering systems based on knowledge graphs are extremely challenging tasks in the field of natural language processing. Most of the existing Chinese Knowledge Base Question Answering(KBQA) can only return the knowledge stored in the knowledge base by extractive methods. Nevertheless, this processing does not conform to the reading habits and cannot solve the Out-of-vocabulary(OOV) problem. In this paper, a new generative question answering method based on knowledge graph is proposed, including three parts of knowledge vocabulary construction, data pre-processing, and answer generation. In the word list construction, BiLSTM-CRF is used to identify the entity in the source text, finding the triples contained in the entity, counting the word frequency, and constructing it. In the part of data pre-processing, a pre-trained language model BERT combining word frequency semantic features is adopted to obtain word vectors. In the answer generation part, one combination of a vocabulary constructed by the knowledge graph and a pointer generator network(PGN) is proposed to point to the corresponding entity for generating answer. The experimental results show that the proposed method can achieve superior performance on WebQA datasets than other methods.

scholarly journals Tackling Graphical Natural Language Processing’s Problems with Recurrent Neural Networks

2019 ◽  
Vol 54 (5) ◽  
Author(s):  
Ali Sami Sosa ◽  
Saja Majeed Mohammed ◽  
Haider Hadi Abbas ◽  
Israa Al Barazanchi
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Question Answering ◽  
Structural Information ◽  
Sequential Data

Recent years have witnessed the success of artificial intelligence–based automated systems that use deep learning, especially recurrent neural network-based models, on many natural language processing problems, including machine translation and question answering. Besides, recurrent neural networks and their variations have been extensively studied with respect to several graph problems and have shown preliminary success. Despite these successes, recurrent neural network -based models continue to suffer from several major drawbacks. First, they can only consume sequential data; thus, linearization is required to serialize input graphs, resulting in the loss of important structural information. In particular, graph nodes that are originally located closely to each other can be very far away after linearization, and this introduces great challenges for recurrent neural networks to model their relation. Second, the serialization results are usually very long, so it takes a long time for recurrent neural networks to encode them. In the methodology of this study, we made the resulting graphs more densely connected so that more useful facts could be inferred, and the problem of graphical natural language processing could be easily decoded with graph recurrent neural network. As a result, the performances with single-typed edges were significantly better than the Local baseline, whereas the combination of all types of edges achieved a much better accuracy than just that of the Local using recurrent neural network. In this paper, we propose a novel graph neural network, named graph recurrent network.

Sign in / Sign up

Export Citation Format

Share Document