An AdaBoost Using a Weak-Learner Generating Several Weak-Hypotheses for Large Training Data of Natural Language Processing

Natural Language Inference (NLI) plays an important role in many natural language processing tasks such as question answering. However, existing NLI modules that are trained on existing NLI datasets have several drawbacks. For example, they do not capture the notion of entity and role well and often end up making mistakes such as “Peter signed a deal” can be inferred from “John signed a deal”. As part of this work, we have developed two datasets that help mitigate such issues and make the systems better at understanding the notion of “entities” and “roles”. After training the existing models on the new dataset we observe that the existing models do not perform well on one of the new benchmark. We then propose a modification to the “word-to-word” attention function which has been uniformly reused across several popular NLI architectures. The resulting models perform as well as their unmodified counterparts on the existing benchmarks and perform significantly well on the new benchmarks that emphasize “roles” and “entities”.

Download Full-text

Learning Structural Kernels for Natural Language Processing

Transactions of the Association for Computational Linguistics ◽

10.1162/tacl_a_00151 ◽

2015 ◽

Vol 3 ◽

pp. 461-473 ◽

Cited By ~ 3

Author(s):

Daniel Beck ◽

Trevor Cohn ◽

Christian Hardmeier ◽

Lucia Specia

Keyword(s):

Natural Language Processing ◽

Model Selection ◽

Natural Language ◽

Language Processing ◽

Bayesian Methods ◽

Training Data ◽

Coarse Grained ◽

Grid Search ◽

Hyperparameter Optimization ◽

Gradient Based

Structural kernels are a flexible learning paradigm that has been widely used in Natural Language Processing. However, the problem of model selection in kernel-based methods is usually overlooked. Previous approaches mostly rely on setting default values for kernel hyperparameters or using grid search, which is slow and coarse-grained. In contrast, Bayesian methods allow efficient model selection by maximizing the evidence on the training data through gradient-based methods. In this paper we show how to perform this in the context of structural kernels by using Gaussian Processes. Experimental results on tree kernels show that this procedure results in better prediction performance compared to hyperparameter optimization via grid search. The framework proposed in this paper can be adapted to other structures besides trees, e.g., strings and graphs, thereby extending the utility of kernel-based methods.

Download Full-text

Deep Persian sentiment analysis: Cross-lingual training for low-resource languages

Journal of Information Science ◽

10.1177/0165551520962781 ◽

2020 ◽

pp. 016555152096278

Author(s):

Rouzbeh Ghasemi ◽

Seyed Arad Ashrafi Asli ◽

Saeedeh Momtazi

Keyword(s):

Natural Language Processing ◽

Natural Language ◽

Sentiment Analysis ◽

Language Processing ◽

Training Data ◽

Target Language ◽

Low Resource ◽

Proposed Model ◽

Significant Difference ◽

Cross Lingual

With the advent of deep neural models in natural language processing tasks, having a large amount of training data plays an essential role in achieving accurate models. Creating valid training data, however, is a challenging issue in many low-resource languages. This problem results in a significant difference between the accuracy of available natural language processing tools for low-resource languages compared with rich languages. To address this problem in the sentiment analysis task in the Persian language, we propose a cross-lingual deep learning framework to benefit from available training data of English. We deployed cross-lingual embedding to model sentiment analysis as a transfer learning model which transfers a model from a rich-resource language to low-resource ones. Our model is flexible to use any cross-lingual word embedding model and any deep architecture for text classification. Our experiments on English Amazon dataset and Persian Digikala dataset using two different embedding models and four different classification networks show the superiority of the proposed model compared with the state-of-the-art monolingual techniques. Based on our experiment, the performance of Persian sentiment analysis improves 22% in static embedding and 9% in dynamic embedding. Our proposed model is general and language-independent; that is, it can be used for any low-resource language, once a cross-lingual embedding is available for the source–target language pair. Moreover, by benefitting from word-aligned cross-lingual embedding, the only required data for a reliable cross-lingual embedding is a bilingual dictionary that is available between almost all languages and the English language, as a potential source language.

Download Full-text

Using distant supervision to augment manually annotated data for relation extraction

10.1101/626226 ◽

2019 ◽

Author(s):

Peng Su ◽

Gang Li ◽

Cathy Wu ◽

K. Vijay-Shanker

Keyword(s):

Deep Learning ◽

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

Relation Extraction ◽

Biomedical Literature ◽

Training Data ◽

Distant Supervision ◽

Large Size ◽

Domain Expertise

AbstractSignificant progress has been made in applying deep learning on natural language processing tasks recently. However, deep learning models typically require a large amount of annotated training data while often only small labeled datasets are available for many natural language processing tasks in biomedical literature. Building large-size datasets for deep learning is expensive since it involves considerable human effort and usually requires domain expertise in specialized fields. In this work, we consider augmenting manually annotated data with large amounts of data using distant supervision. However, data obtained by distant supervision is often noisy, we first apply some heuristics to remove some of the incorrect annotations. Then using methods inspired from transfer learning, we show that the resulting models outperform models trained on the original manually annotated sets.

Download Full-text

Database Tuning using Natural Language Processing

ACM SIGMOD Record ◽

10.1145/3503780.3503788 ◽

2021 ◽

Vol 50 (3) ◽

pp. 27-28

Author(s):

Immanuel Trummer

Keyword(s):

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

Training Data ◽

Language Models ◽

Learning Approaches ◽

Training Samples ◽

Starting Point ◽

Training Cost ◽

Transformer Model

Introduction. We have seen significant advances in the state of the art in natural language processing (NLP) over the past few years [20]. These advances have been driven by new neural network architectures, in particular the Transformer model [19], as well as the successful application of transfer learning approaches to NLP [13]. Typically, training for specific NLP tasks starts from large language models that have been pre-trained on generic tasks (e.g., predicting obfuscated words in text [5]) for which large amounts of training data are available. Using such models as a starting point reduces task-specific training cost as well as the number of required training samples by orders of magnitude [7]. These advances motivate new use cases for NLP methods in the context of databases.

Download Full-text

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Frontiers in Human Neuroscience ◽

10.3389/fnhum.2021.659410 ◽

2021 ◽

Vol 15 ◽

Author(s):

Nora Hollenstein ◽

Cedric Renggli ◽

Benjamin Glaus ◽

Maria Barrett ◽

Marius Troendle ◽

...

Keyword(s):

Machine Learning ◽

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

Large Scale ◽

Brain Activity ◽

Training Data ◽

Human Cognition ◽

Special Focus ◽

Eeg Data

Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available.

Download Full-text

Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance

Journal of Medical Systems ◽

10.1007/s10916-021-01761-4 ◽

2021 ◽

Vol 45 (10) ◽

Author(s):

A. W. Olthof ◽

P. M. A. van Ooijen ◽

L. J. Cornelissen

Keyword(s):

Neural Network ◽

Deep Learning ◽

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

Predictive Value ◽

Training Data ◽

Radiology Reports ◽

Dataset Size ◽

The Impact

AbstractIn radiology, natural language processing (NLP) allows the extraction of valuable information from radiology reports. It can be used for various downstream tasks such as quality improvement, epidemiological research, and monitoring guideline adherence. Class imbalance, variation in dataset size, variation in report complexity, and algorithm type all influence NLP performance but have not yet been systematically and interrelatedly evaluated. In this study, we investigate these factors on the performance of four types [a fully connected neural network (Dense), a long short-term memory recurrent neural network (LSTM), a convolutional neural network (CNN), and a Bidirectional Encoder Representations from Transformers (BERT)] of deep learning-based NLP. Two datasets consisting of radiologist-annotated reports of both trauma radiographs (n = 2469) and chest radiographs and computer tomography (CT) studies (n = 2255) were split into training sets (80%) and testing sets (20%). The training data was used as a source to train all four model types in 84 experiments (Fracture-data) and 45 experiments (Chest-data) with variation in size and prevalence. The performance was evaluated on sensitivity, specificity, positive predictive value, negative predictive value, area under the curve, and F score. After the NLP of radiology reports, all four model-architectures demonstrated high performance with metrics up to > 0.90. CNN, LSTM, and Dense were outperformed by the BERT algorithm because of its stable results despite variation in training size and prevalence. Awareness of variation in prevalence is warranted because it impacts sensitivity and specificity in opposite directions.

Download Full-text

Enriching Word Vectors with Subword Information

Transactions of the Association for Computational Linguistics ◽

10.1162/tacl_a_00051 ◽

2017 ◽

Vol 5 ◽

pp. 135-146 ◽

Cited By ~ 1156

Author(s):

Piotr Bojanowski ◽

Edouard Grave ◽

Armand Joulin ◽

Tomas Mikolov

Keyword(s):

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

State Of The Art ◽

Training Data ◽

Vector Representation ◽

New Approach ◽

Word Similarity ◽

Art Performance ◽

N Gram

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

Download Full-text

A Skin Cancer Detection Interactive Application Based on CNN and NLP

Journal of Physics Conference Series ◽

10.1088/1742-6596/2078/1/012036 ◽

2021 ◽

Vol 2078 (1) ◽

pp. 012036

Author(s):

Xuping Gong ◽

Yuting Xiao

Keyword(s):

Neural Network ◽

Natural Language Processing ◽

Skin Cancer ◽

Natural Language ◽

Language Processing ◽

Cancer Diagnosis ◽

Training Data ◽

Interactive Network ◽

Questions And Answers ◽

Interactive Application

Abstract Skin cancer is the most common cancer with several different types. According to current estimations, one in five Americans will develop skin cancer in their lifetime. Therefore, early diagnosis and treatment of it is of crucial significance. Several advanced image processing methods have been applied to predict skin cancer. However, few researchers utilize those methods to build an interactive application. In this work, we implemented an interactive skin cancer diagnosis website, combining the convolutional neural network (CNN) and natural language processing (NLP) technology. The neural network model uses four convolutional layers and dense layers respectively to improve the accuracy. Two max-pooling layers were used to reduce redundant information. To address the severe overfitting problem, we chose to utilize the batch normalization along with dropout layers. Based on our results, 0.9935 in accuracy and 0.0225 loss is realized for training data, and accuracy of 0.8393 and 0.6648 loss for testing data. Natural language processing (NLP) was used to implement a chatbot for interaction with users. We crawled skin cancer related questions and answers from Quora and used them to train our chatbot. Lastly, we combined CNN and NLP to build an interactive skin cancer diagnosis website. VUE.js and Django were used to build the front-end and back-end of our website. These results offer a guideline for combining artificial intelligence with not only medicine but also interactive network, which enables people to get medical care more easily.

Download Full-text

Weakly Supervised Learning for Categorization of Medical Inquiries for Customer Service Effectiveness

Frontiers in Research Metrics and Analytics ◽

10.3389/frma.2021.683400 ◽

2021 ◽

Vol 6 ◽

Author(s):

Shikha Singhal ◽

Bharat Hegde ◽

Prathamesh Karmalkar ◽

Justna Muhith ◽

Harsha Gurulingappa

Keyword(s):

Machine Learning ◽

Natural Language Processing ◽

Natural Language ◽

Supervised Learning ◽

Language Processing ◽

Medical Information ◽

Service Providers ◽

Training Data ◽

Weakly Supervised Learning ◽

Weakly Supervised

With the growing unstructured data in healthcare and pharmaceutical, there has been a drastic adoption of natural language processing for generating actionable insights from text data sources. One of the key areas of our exploration is the Medical Information function within our organization. We receive a significant amount of medical information inquires in the form of unstructured text. An enterprise-level solution must deal with medical information interactions via multiple communication channels which are always nuanced with a variety of keywords and emotions that are unique to the pharmaceutical industry. There is a strong need for an effective solution to leverage the contextual knowledge of the medical information business along with digital tenants of natural language processing (NLP) and machine learning to build an automated and scalable process that generates real-time insights on conversation categories. The traditional supervised learning methods rely on a huge set of manually labeled training data and this dataset is difficult to attain due to high labeling costs. Thus, the solution is incomplete without its ability to self-learn and improve. This necessitates techniques to automatically build relevant training data using a weakly supervised approach from textual inquiries across consumers, healthcare professionals, sales, and service providers. The solution has two fundamental layers of NLP and machine learning. The first layer leverages heuristics and knowledgebase to identify the potential categories and build an annotated training data. The second layer, based on machine learning and deep learning, utilizes the training data generated using the heuristic approach for identifying categories and sub-categories associated with verbatim. Here, we present a novel approach harnessing the power of weakly supervised learning combined with multi-class classification for improved categorization of medical information inquiries.

Download Full-text