scholarly journals Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

2020 ◽  
Author(s):  
Trapit Bansal ◽  
Rishikesh Jha ◽  
Tsendsuren Munkhdalai ◽  
Andrew McCallum
Keyword(s):  
Natural Language ◽  
Meta Learning ◽  
Classification Tasks ◽  
Language Classification

scholarly journals Toward a clinical text encoder: pretraining for clinical natural language processing with applications to substance misuse

2019 ◽  
Vol 26 (11) ◽  
pp. 1272-1278 ◽  
Author(s):  
Dmitriy Dligach ◽  
Majid Afshar ◽  
Timothy Miller
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Substance Misuse ◽  
Research Direction ◽  
Clinical Text ◽  
Universal Properties ◽  
Classification Tasks ◽  
Performance Gains ◽  
Clinical Natural Language Processing

Abstract Objective Our objective is to develop algorithms for encoding clinical text into representations that can be used for a variety of phenotyping tasks. Materials and Methods Obtaining large datasets to take advantage of highly expressive deep learning methods is difficult in clinical natural language processing (NLP). We address this difficulty by pretraining a clinical text encoder on billing code data, which is typically available in abundance. We explore several neural encoder architectures and deploy the text representations obtained from these encoders in the context of clinical text classification tasks. While our ultimate goal is learning a universal clinical text encoder, we also experiment with training a phenotype-specific encoder. A universal encoder would be more practical, but a phenotype-specific encoder could perform better for a specific task. Results We successfully train several clinical text encoders, establish a new state-of-the-art on comorbidity data, and observe good performance gains on substance misuse data. Discussion We find that pretraining using billing codes is a promising research direction. The representations generated by this type of pretraining have universal properties, as they are highly beneficial for many phenotyping tasks. Phenotype-specific pretraining is a viable route for trading the generality of the pretrained encoder for better performance on a specific phenotyping task. Conclusions We successfully applied our approach to many phenotyping tasks. We conclude by discussing potential limitations of our approach.

scholarly journals Variance-reduced First-order Meta-learning for Natural Language Processing Tasks

2021 ◽  
Author(s):  
Lingxiao Wang ◽  
Kevin Huang ◽  
Tengyu Ma ◽  
Quanquan Gu ◽  
Jing Huang
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
First Order ◽  
Meta Learning

Counterfactual Generative Smoothing for Imbalanced Natural Language Classification

2021 ◽  
Author(s):  
Hojae Han ◽  
Seungtaek Choi ◽  
Myeongho Jeong ◽  
Jin-woo Park ◽  
Seung-won Hwang
Keyword(s):  
Natural Language ◽  
Language Classification

Language Classification and Recognition From Audio Using Deep Belief Network

2021 ◽  
pp. 189-213
Author(s):  
Santhi Selvaraj ◽  
Raja Sekar J. ◽  
Amutha S.
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Markov Models ◽  
Learning Algorithm ◽  
Back Propagation ◽  
Gaussian Mixture ◽  
Deep Belief Network ◽  
Mel Frequency Cepstral Coefficients ◽  
Belief Network ◽  
Language Classification

The main objective is to recognize the chat from social media as spoken language by using deep belief network (DBN). Currently, language classification is one of the main applications of natural language processing, artificial intelligence, and deep learning. Language classification is the process of ascertaining the information being presented in which natural language and recognizing a language from the audio sound. Presently, most language recognition systems are based on hidden Markov models and Gaussian mixture models that support both acoustic and sequential modeling. This chapter presents a DBN-based recognition system in three different languages, namely English, Hindi, and Tamil. The evaluation of languages is performed on the self built recorded database, which extracts the mel-frequency cepstral coefficients features from the speeches. These features are fed into the DBN with a back propagation learning algorithm for the recognition process. Accuracy of the recognition is efficient for the chosen languages and the system performance is assessed on three different languages.

scholarly journals Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems

2019 ◽  
Author(s):  
Fei Mi ◽  
Minlie Huang ◽  
Jiyong Zhang ◽  
Boi Faltings
Keyword(s):  
Natural Language ◽  
Natural Language Generation ◽  
Dialogue Systems ◽  
Language Generation ◽  
Low Resource ◽  
Recent Success ◽  
Resource Setting ◽  
Meta Learning ◽  
Training Examples ◽  
Task Oriented

Natural language generation (NLG) is an essential component of task-oriented dialogue systems. Despite the recent success of neural approaches for NLG, they are typically developed for particular domains with rich annotated training examples. In this paper, we study NLG in a low-resource setting to generate sentences in new scenarios with handful training examples. We formulate the problem from a meta-learning perspective, and propose a generalized optimization-based approach (Meta-NLG) based on the well-recognized model-agnostic meta-learning (MAML) algorithm. Meta-NLG defines a set of meta tasks, and directly incorporates the objective of adapting to new low-resource NLG tasks into the meta-learning optimization process. Extensive experiments are conducted on a large multi-domain dataset (MultiWoz) with diverse linguistic variations. We show that Meta-NLG significantly outperforms other training procedures in various low-resource configurations. We analyze the results, and demonstrate that Meta-NLG adapts extremely fast and well to low-resource situations.

Sign in / Sign up

Export Citation Format

Share Document