scholarly journals Impact of Using Bidirectional Encoder Representations from Transformers (BERT) Models for Arabic Dialogue Acts Identification

2021 ◽  
Vol 26 (5) ◽  
pp. 469-475
Author(s):  
Alaa Joukhadar ◽  
Nada Ghneim ◽  
Ghaida Rebdawi
Keyword(s):  
Language Processing ◽  
State Of The Art ◽  
Language Model ◽  
Correct Identification ◽  
Dialogue Systems ◽  
Morphological Segmentation ◽  
Representation Model ◽  
Dialogue Acts ◽  
Language Representation ◽  
The Impact

In Human-Computer dialogue systems, the correct identification of the intent underlying a speaker's utterance is crucial to the success of a dialogue. Several researches have studied the Dialogue Act Classification (DAC) task to identify Dialogue Acts (DA) for different languages. Recently, the emergence of Bidirectional Encoder Representations from Transformers (BERT) models, enabled establishing state-of-the-art results for a variety of natural language processing tasks in different languages. Very few researches have been done in the Arabic Dialogue acts identification task. The BERT representation model has not been studied yet in Arabic Dialogue acts detection task. In this paper, we propose a model using BERT language representation to identify Arabic Dialogue Acts. We explore the impact of using different BERT models: AraBERT Original (v0.1, v1), AraBERT Base (v0.2, and v2) and AraBERT Large (v0.2, and v2), which are pretrained on different Arabic corpora (different in size, morphological segmentation, language model window, …). The comparison was performed on two available Arabic datasets. Using AraBERTv0.2-base model for dialogue representations outperformed all other pretrained models. Moreover, we compared the performance of AraBERTv0.2-base model to the state-of-the-art approaches applied on the two datasets. The comparison showed that this representation model outperformed the performance both state-of-the-art models.

scholarly journals Enhancing clinical concept extraction with contextual embeddings

2019 ◽  
Vol 26 (11) ◽  
pp. 1297-1304 ◽  
Author(s):  
Yuqi Si ◽  
Jingqi Wang ◽  
Hua Xu ◽  
Kirk Roberts
Keyword(s):  
Language Processing ◽  
Semantic Information ◽  
Medical Information ◽  
State Of The Art ◽  
Language Model ◽  
The State ◽  
Concept Extraction ◽  
The Impact ◽  
Clinical Concept ◽  
Embedding Methods

Abstract Objective Neural network–based representations (“embeddings”) have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (eg, ELMo, BERT) have further pushed the state of the art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). Materials and Methods Both off-the-shelf, open-domain embeddings and pretrained clinical embeddings from MIMIC-III (Medical Information Mart for Intensive Care III) are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings and compare these on 4 concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pretraining time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings. Results Contextual embeddings pretrained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65. Conclusions We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate that contextual embeddings encode valuable semantic information not accounted for in traditional word representations.

scholarly journals A Pre-Training Based Personalized Dialogue Generation Model with Persona-Sparse Data

2020 ◽  
Vol 34 (05) ◽  
pp. 9693-9700
Author(s):  
Yinhe Zheng ◽  
Rongsheng Zhang ◽  
Minlie Huang ◽  
Xiaoxi Mao
Keyword(s):  
State Of The Art ◽  
Language Model ◽  
Sparse Data ◽  
Generation Model ◽  
Dialogue Systems ◽  
Inference Process ◽  
Training Process ◽  
Natural Languages ◽  
Dialogue Model ◽  
Proposed Model

Endowing dialogue systems with personas is essential to deliver more human-like conversations. However, this problem is still far from well explored due to the difficulties of both embodying personalities in natural languages and the persona sparsity issue observed in most dialogue corpora. This paper proposes a pre-training based personalized dialogue model that can generate coherent responses using persona-sparse dialogue data. In this method, a pre-trained language model is used to initialize an encoder and decoder, and personal attribute embeddings are devised to model richer dialogue contexts by encoding speakers' personas together with dialogue histories. Further, to incorporate the target persona in the decoding process and to balance its contribution, an attention routing structure is devised in the decoder to merge features extracted from the target persona and dialogue contexts using dynamically predicted weights. Our model can utilize persona-sparse dialogues in a unified manner during the training process, and can also control the amount of persona-related features to exhibit during the inference process. Both automatic and manual evaluation demonstrates that the proposed model outperforms state-of-the-art methods for generating more coherent and persona consistent responses with persona-sparse data.

scholarly journals Leveraging Linguistic Coordination in Reranking N-Best Candidates For End-to-End Response Selection Using BERT

2021 ◽  
Vol 34 (1) ◽  
Author(s):  
Mingzhi Yu ◽  
Diane Litman
Keyword(s):  
Response Selection ◽  
State Of The Art ◽  
Language Model ◽  
Linguistic Theory ◽  
Dialogue Systems ◽  
Best Response ◽  
End To End ◽  
Selection Tasks

Retrieval-based dialogue systems select the best response from many candidates. Although many state-of-the-art models have shown promising performance in dialogue response selection tasks, there is still quite a gap between R@1 and R@10 performance. To address this, we propose to leverage linguistic coordination (a phenomenon that individuals tend to develop similar linguistic behaviors in conversation) to rerank the N-best candidates produced by BERT, a state-of-the-art pre-trained language model. Our results show an improvement in R@1 compared to BERT baselines, demonstrating the utility of repairing machine-generated outputs by leveraging a linguistic theory.

scholarly journals FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

2020 ◽  
Author(s):  
Zhuang Liu ◽  
Degen Huang ◽  
Kaiyu Huang ◽  
Zhuang Li ◽  
Jun Zhao
Keyword(s):  
Deep Learning ◽  
Text Mining ◽  
Language Processing ◽  
Large Scale ◽  
Language Model ◽  
Training Data ◽  
Domain Specific ◽  
Current State ◽  
Language Representation ◽  
Financial Domain

There is growing interest in the tasks of financial text mining. Over the past few years, the progress of Natural Language Processing (NLP) based on deep learning advanced rapidly. Significant progress has been made with deep learning showing promising results on financial text mining models. However, as NLP models require large amounts of labeled training data, applying deep learning to financial text mining is often unsuccessful due to the lack of labeled training data in financial fields. To address this issue, we present FinBERT (BERT for Financial Text Mining) that is a domain specific language model pre-trained on large-scale financial corpora. In FinBERT, different from BERT, we construct six pre-training tasks covering more knowledge, simultaneously trained on general corpora and financial domain corpora, which can enable FinBERT model better to capture language knowledge and semantic information. The results show that our FinBERT outperforms all current state-of-the-art models. Extensive experimental results demonstrate the effectiveness and robustness of FinBERT. The source code and pre-trained models of FinBERT are available online.

Making the most of MOGUL: Reflections on interlanguage in childhood language disorders

2017 ◽  
Vol 55 (4) ◽  
pp. 349-364
Author(s):  
Susan H. Foster-Cohen
Keyword(s):  
Language Processing ◽  
Language Change ◽  
Language Disorder ◽  
Language Model ◽  
Language Disorders ◽  
Relevance Theory ◽  
Psychological Resources ◽  
Wide Range ◽  
On Line ◽  
The Impact

Abstract Interlanguage is a concept that is manifest in any trajectory of language change in a learner: typical first language, second language or language disorder. To understand those trajectories we need a rich psychological model of what creates them. This paper applies one such model–Sharwood Smith and Truscott’s Modular On-Line Growth and Use of Language model (MOGUL)–to childhood developmental language disorders, and suggests that the model’s components of language processing and their interaction shed significant light on why children with a wide range of different language disorders exhibit the language behaviours that are the characteristic of their diagnoses. Following a brief summary of the MOGUL model, the paper explores the impact on language development of differences in sensori-motor input, in the functioning of the various modules and the interfaces between them, and in the storage and activation of memory. Like Relevance Theory, with which the paper makes a direct connection, MOGUL encourages one to take a view of developmental language disorder as emerging from the same set of psychological resources as typical development (first or second) but as being the result of adjustments to, or compensations for, differences in how the various modules function and connect with each other.

scholarly journals BioBERT: a pre-trained biomedical language representation model for biomedical text mining

2019 ◽  
Author(s):  
Jinhyuk Lee ◽  
Wonjin Yoon ◽  
Sungdong Kim ◽  
Donghyeon Kim ◽  
Sunkyu Kim ◽  
...  
Keyword(s):  
Text Mining ◽  
State Of The Art ◽  
Biomedical Literature ◽  
Fine Tuning ◽  
Biomedical Text ◽  
Biomedical Text Mining ◽  
Representation Model ◽  
Score Improvement ◽  
Language Representation ◽  
Previous State

Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

scholarly journals P2V-MAP: Mapping Market Structures for Large Retail Assortments

2019 ◽  
Vol 56 (4) ◽  
pp. 557-580 ◽  
Author(s):  
Sebastian Gabel ◽  
Daniel Guhl ◽  
Daniel Klapper
Keyword(s):  
Language Processing ◽  
State Of The Art ◽  
A Priori ◽  
Language Model ◽  
Market Structures ◽  
Econometric Methods ◽  
Exploratory Approach ◽  
Using Data ◽  
Network Language ◽  
Product Map

The authors propose a new, exploratory approach for analyzing market structures that leverages two recent methodological advances in natural language processing and machine learning. They customize a neural network language model to derive latent product attributes by analyzing the co-occurrences of products in shopping baskets. Applying dimensionality reduction to the latent attributes yields a two-dimensional product map. This method is well-suited to retailers because it relies on data that are readily available from their checkout systems and facilitates their analyses of cross-category product complementarity, in addition to within-category substitution. The approach has high usability because it is automated, is scalable and does not require a priori assumptions. Its results are easy to interpret and update as new market basket data are collected. The authors validate their approach both by conducting an extensive simulation study and by comparing their results with those of state-of-the-art, econometric methods for modeling product relationships. The application of this approach using data collected at a leading German grocery retailer underlines its usefulness and provides novel findings that are relevant to assortment-related decisions.

scholarly journals Active Learning for Effectively Fine-Tuning Transfer Learning to Downstream Task

2021 ◽  
Vol 12 (2) ◽  
pp. 1-24
Author(s):  
Md Abul Bashar ◽  
Richi Nayak
Keyword(s):  
Active Learning ◽  
Transfer Learning ◽  
Language Processing ◽  
State Of The Art ◽  
Language Model ◽  
Ensemble Classifier ◽  
Classification Performance ◽  
Fine Tuning ◽  
Linguistic Features ◽  
Better Than

Language model (LM) has become a common method of transfer learning in Natural Language Processing (NLP) tasks when working with small labeled datasets. An LM is pretrained using an easily available large unlabelled text corpus and is fine-tuned with the labelled data to apply to the target (i.e., downstream) task. As an LM is designed to capture the linguistic aspects of semantics, it can be biased to linguistic features. We argue that exposing an LM model during fine-tuning to instances that capture diverse semantic aspects (e.g., topical, linguistic, semantic relations) present in the dataset will improve its performance on the underlying task. We propose a Mixed Aspect Sampling (MAS) framework to sample instances that capture different semantic aspects of the dataset and use the ensemble classifier to improve the classification performance. Experimental results show that MAS performs better than random sampling as well as the state-of-the-art active learning models to abuse detection tasks where it is hard to collect the labelled data for building an accurate classifier.

scholarly journals Evaluation of the Coherence of Polish Texts Using Neural Network Models

2021 ◽  
Vol 11 (7) ◽  
pp. 3210
Author(s):  
Sergii Telenyk ◽  
Sergiy Pogorilyy ◽  
Artem Kramov
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Word Order ◽  
State Of The Art ◽  
Network Models ◽  
Neural Network Models ◽  
Similarity Graph ◽  
Coherence Estimation ◽  
The Impact ◽  
Machine Learning Models

Coherence evaluation of texts falls into a category of natural language processing tasks. The evaluation of texts’ coherence implies the estimation of their semantic and logical integrity; such a feature of a text can be utilized during the solving of multidisciplinary tasks (SEO analysis, medicine area, detection of fake texts, etc.). In this paper, different state-of-the-art coherence evaluation methods based on machine learning models have been analyzed. The investigation of the effectiveness of different methods for the coherence estimation of Polish texts has been performed. The impact of text’s features on the output coherence value has been analyzed using different approaches of a semantic similarity graph. Two neural networks based on LSTM layers and a pre-trained BERT model correspondingly have been designed and trained for the coherence estimation of input texts. The results obtained may indicate that both lexical and semantic components should be taken into account during the coherence evaluation of Polish documents; moreover, it is advisable to analyze corresponding documents in a sentence-by-sentence manner taking into account word order. According to the retrieved accuracy of the proposed neural networks, it can be concluded that suggested models may be used in order to solve typical coherence estimation tasks for a Polish corpus.

scholarly journals CloudLM: a Cloud-based Language Model for Machine Translation

2016 ◽  
Vol 105 (1) ◽  
pp. 51-61 ◽  
Author(s):  
Jorge Ferrández-Tordera ◽  
Sergio Ortiz-Rojas ◽  
Antonio Toral
Keyword(s):  
Big Data ◽  
Natural Language Processing ◽  
Machine Translation ◽  
Language Processing ◽  
State Of The Art ◽  
Language Model ◽  
Essential Element ◽  
Language Models ◽  
Language Modelling ◽  
Statistical Approaches

Abstract Language models (LMs) are an essential element in statistical approaches to natural language processing for tasks such as speech recognition and machine translation (MT). The advent of big data leads to the availability of massive amounts of data to build LMs, and in fact, for the most prominent languages, using current techniques and hardware, it is not feasible to train LMs with all the data available nowadays. At the same time, it has been shown that the more data is used for a LM the better the performance, e.g. for MT, without any indication yet of reaching a plateau. This paper presents CloudLM, an open-source cloud-based LM intended for MT, which allows to query distributed LMs. CloudLM relies on Apache Solr and provides the functionality of state-of-the-art language modelling (it builds upon KenLM), while allowing to query massive LMs (as the use of local memory is drastically reduced), at the expense of slower decoding speed.

Sign in / Sign up

Export Citation Format

Share Document