scholarly journals Re-framing Incremental Deep Language Models for Dialogue Processing with Multi-task Learning

2020 ◽  
Author(s):  
Morteza Rohanian ◽  
Julian Hough
Keyword(s):  
Language Models ◽  
Task Learning

scholarly journals Multi-Task Learning for Knowledge Graph Completion with Pre-trained Language Models

2020 ◽  
Author(s):  
Bosung Kim ◽  
Taesuk Hong ◽  
Youngjoong Ko ◽  
Jungyun Seo
Keyword(s):  
Language Models ◽  
Knowledge Graph ◽  
Task Learning

scholarly journals An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models

2020 ◽  
Vol 8 ◽  
pp. 621-633
Author(s):  
Lifu Tu ◽  
Garima Lalwani ◽  
Spandana Gella ◽  
He He
Keyword(s):  
Empirical Study ◽  
Natural Language ◽  
Recent Work ◽  
Language Models ◽  
Task Learning ◽  
The Right

Recent work has shown that pre-trained language models such as BERT improve robustness to spurious correlations in the dataset. Intrigued by these results, we find that the key to their success is generalization from a small amount of counterexamples where the spurious correlations do not hold. When such minority examples are scarce, pre-trained models perform as poorly as models trained from scratch. In the case of extreme minority, we propose to use multi-task learning (MTL) to improve generalization. Our experiments on natural language inference and paraphrase identification show that MTL with the right auxiliary tasks significantly improves performance on challenging examples without hurting the in-distribution performance. Further, we show that the gain from MTL mainly comes from improved generalization from the minority examples. Our results highlight the importance of data diversity for overcoming spurious correlations. 1

scholarly journals Multi-task Learning Based Online Dialogic Instruction Detection with Pre-trained Language Models

2021 ◽  
pp. 183-189
Author(s):  
Yang Hao ◽  
Hang Li ◽  
Wenbiao Ding ◽  
Zhongqin Wu ◽  
Jiliang Tang ◽  
...  
Keyword(s):  
Language Models ◽  
Dialogic Instruction ◽  
Task Learning

scholarly journals Identification of Semantically Similar Sentences in Clinical Notes: Iterative Intermediate Training Using Multi-Task Learning (Preprint)

2020 ◽  
Author(s):  
Diwakar Mahajan ◽  
Ananya Poddar ◽  
Jennifer J Liang ◽  
Yen-Ting Lin ◽  
John M Prager ◽  
...  
Keyword(s):  
Text Mining ◽  
Semantic Similarity ◽  
Language Models ◽  
Data Set ◽  
Clinical Text ◽  
Clinical Notes ◽  
Task Learning ◽  
Training Approach ◽  
Clinical Domain ◽  
Semantic Textual Similarity

BACKGROUND Although electronic health records (EHRs) have been widely adopted in health care, effective use of EHR data is often limited because of redundant information in clinical notes introduced by the use of templates and copy-paste during note generation. Thus, it is imperative to develop solutions that can condense information while retaining its value. A step in this direction is measuring the semantic similarity between clinical text snippets. To address this problem, we participated in the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing Consortium (OHNLP) clinical semantic textual similarity (ClinicalSTS) shared task. OBJECTIVE This study aims to improve the performance and robustness of semantic textual similarity in the clinical domain by leveraging manually labeled data from related tasks and contextualized embeddings from pretrained transformer-based language models. METHODS The ClinicalSTS data set consists of 1642 pairs of deidentified clinical text snippets annotated in a continuous scale of 0-5, indicating degrees of semantic similarity. We developed an iterative intermediate training approach using multi-task learning (IIT-MTL), a multi-task training approach that employs iterative data set selection. We applied this process to bidirectional encoder representations from transformers on clinical text mining (ClinicalBERT), a pretrained domain-specific transformer-based language model, and fine-tuned the resulting model on the target ClinicalSTS task. We incrementally ensembled the output from applying IIT-MTL on ClinicalBERT with the output of other language models (bidirectional encoder representations from transformers for biomedical text mining [BioBERT], multi-task deep neural networks [MT-DNN], and robustly optimized BERT approach [RoBERTa]) and handcrafted features using regression-based learning algorithms. On the basis of these experiments, we adopted the top-performing configurations as our official submissions. RESULTS Our system ranked first out of 87 submitted systems in the 2019 n2c2/OHNLP ClinicalSTS challenge, achieving state-of-the-art results with a Pearson correlation coefficient of 0.9010. This winning system was an ensembled model leveraging the output of IIT-MTL on ClinicalBERT with BioBERT, MT-DNN, and handcrafted medication features. CONCLUSIONS This study demonstrates that IIT-MTL is an effective way to leverage annotated data from related tasks to improve performance on a target task with a limited data set. This contribution opens new avenues of exploration for optimized data set selection to generate more robust and universal contextual representations of text in the clinical domain.

scholarly journals Identification of Semantically Similar Sentences in Clinical Notes: Iterative Intermediate Training Using Multi-Task Learning

10.2196/22508 ◽  
2020 ◽  
Vol 8 (11) ◽  
pp. e22508
Author(s):  
Diwakar Mahajan ◽  
Ananya Poddar ◽  
Jennifer J Liang ◽  
Yen-Ting Lin ◽  
John M Prager ◽  
...  
Keyword(s):  
Text Mining ◽  
Semantic Similarity ◽  
Language Models ◽  
Data Set ◽  
Clinical Text ◽  
Clinical Notes ◽  
Task Learning ◽  
Training Approach ◽  
Clinical Domain ◽  
Semantic Textual Similarity

Background Although electronic health records (EHRs) have been widely adopted in health care, effective use of EHR data is often limited because of redundant information in clinical notes introduced by the use of templates and copy-paste during note generation. Thus, it is imperative to develop solutions that can condense information while retaining its value. A step in this direction is measuring the semantic similarity between clinical text snippets. To address this problem, we participated in the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing Consortium (OHNLP) clinical semantic textual similarity (ClinicalSTS) shared task. Objective This study aims to improve the performance and robustness of semantic textual similarity in the clinical domain by leveraging manually labeled data from related tasks and contextualized embeddings from pretrained transformer-based language models. Methods The ClinicalSTS data set consists of 1642 pairs of deidentified clinical text snippets annotated in a continuous scale of 0-5, indicating degrees of semantic similarity. We developed an iterative intermediate training approach using multi-task learning (IIT-MTL), a multi-task training approach that employs iterative data set selection. We applied this process to bidirectional encoder representations from transformers on clinical text mining (ClinicalBERT), a pretrained domain-specific transformer-based language model, and fine-tuned the resulting model on the target ClinicalSTS task. We incrementally ensembled the output from applying IIT-MTL on ClinicalBERT with the output of other language models (bidirectional encoder representations from transformers for biomedical text mining [BioBERT], multi-task deep neural networks [MT-DNN], and robustly optimized BERT approach [RoBERTa]) and handcrafted features using regression-based learning algorithms. On the basis of these experiments, we adopted the top-performing configurations as our official submissions. Results Our system ranked first out of 87 submitted systems in the 2019 n2c2/OHNLP ClinicalSTS challenge, achieving state-of-the-art results with a Pearson correlation coefficient of 0.9010. This winning system was an ensembled model leveraging the output of IIT-MTL on ClinicalBERT with BioBERT, MT-DNN, and handcrafted medication features. Conclusions This study demonstrates that IIT-MTL is an effective way to leverage annotated data from related tasks to improve performance on a target task with a limited data set. This contribution opens new avenues of exploration for optimized data set selection to generate more robust and universal contextual representations of text in the clinical domain.

Adolescent Language: Models, Assessment, and Links to Reading

2019 ◽  
Author(s):  
Amanda Goodwin ◽  
Yaacov Petscher ◽  
Jamie Tock
Keyword(s):  
Reading Comprehension ◽  
Bifactor Model ◽  
Language Models ◽  
Multiple Group ◽  
Global Factor ◽  
Eighth Grade Students ◽  
Key Aspects ◽  
Future Work ◽  
The Relationship ◽  
Best Fit

Various models have highlighted the complexity of language. Building on foundational ideas regarding three key aspects of language, our study contributes to the literature by 1) exploring broader conceptions of morphology, vocabulary, and syntax, 2) operationalizing this theoretical model into a gamified, standardized, computer-adaptive assessment of language for fifth to eighth grade students entitled Monster, PI, and 3) uncovering further evidence regarding the relationship between language and standardized reading comprehension via this assessment. Multiple-group item response theory (IRT) across grades show that morphology was best fit by a bifactor model of task specific factors along with a global factor related to each skill. Vocabulary was best fit by a bifactor model that identifies performance overall and on specific words. Syntax, though, was best fit by a unidimensional model. Next, Monster, PI produced reliable scores suggesting language can be assessed efficiently and precisely for students via this model. Lastly, performance on Monster, PI explained more than 50% of variance in standardized reading, suggesting operationalizing language via Monster, PI can provide meaningful understandings of the relationship between language and reading comprehension. Specifically, considering just a subset of a construct, like identification of units of meaning, explained significantly less variance in reading comprehension. This highlights the importance of considering these broader constructs. Implications indicate that future work should consider a model of language where component areas are considered broadly and contributions to reading comprehension are explored via general performance on components as well as skill level performance.

Backwards Fading to Speed Task Learning

2013 ◽  
Author(s):  
Peter S. Schaefer ◽  
Clinton R. Irvin ◽  
Paul N. Blankenbeckler ◽  
C. J. Brogdon
Keyword(s):  
Task Learning ◽  
Speed Task

Multi-Task Learning Using Mismatched Transcription for Under-Resourced Speech Recognition

2017 ◽  
Author(s):  
Van Hai Do ◽  
Nancy F. Chen ◽  
Boon Pang Lim ◽  
Mark Hasegawa-Johnson
Keyword(s):  
Speech Recognition ◽  
Task Learning
Sign in / Sign up

Export Citation Format

Share Document