Joint Representations of Texts and Labels with Compositional Loss for Short Text Classification

Journal of Web Engineering ◽

10.13052/jwe1540-9589.2035 ◽

2021 ◽

Author(s):

Ming Hao ◽

Weijing Wang ◽

Fang Zhou

Keyword(s):

Language Processing ◽

Text Classification ◽

Ground Truth ◽

Language Models ◽

Text Representation ◽

Short Text ◽

Practical Applications ◽

Classical Models ◽

Multi Class Classification ◽

Public Datasets

Short text classification is an important foundation for natural language processing (NLP) tasks. Though, the text classification based on deep language models (DLMs) has made a significant headway, in practical applications however, some texts are ambiguous and hard to classify in multi-class classification especially, for short texts whose context length is limited. The mainstream method improves the distinction of ambiguous text by adding context information. However, these methods rely only the text representation, and ignore that the categories overlap and are not completely independent of each other. In this paper, we establish a new general method to solve the problem of ambiguous text classification by introducing label embedding to represent each category, which makes measurable difference between the categories. Further, a new compositional loss function is proposed to train the model, which makes the text representation closer to the ground-truth label and farther away from others. Finally, a constraint is obtained by calculating the similarity between the text representation and label embedding. Errors caused by ambiguous text can be corrected by adding constraints to the output layer of the model. We apply the method to three classical models and conduct experiments on six public datasets. Experiments show that our method can effectively improve the classification accuracy of the ambiguous texts. In addition, combining our method with BERT, we obtain the state-of-the-art results on the CNT dataset.

Deep Short Text Classification with Knowledge Powered Attention

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33016252 ◽

2019 ◽

Vol 33 ◽

pp. 6252-6259 ◽

Cited By ~ 11

Author(s):

Jindong Chen ◽

Yizhou Hu ◽

Jingping Liu ◽

Yanghua Xiao ◽

Haiyun Jiang

Keyword(s):

Language Processing ◽

Text Classification ◽

Deep Neural Networks ◽

Contextual Information ◽

Semantic Representation ◽

Training Data ◽

Conceptual Information ◽

Short Text ◽

Public Datasets ◽

Traditional Approaches

Short text classification is one of important tasks in Natural Language Processing (NLP). Unlike paragraphs or documents, short texts are more ambiguous since they have not enough contextual information, which poses a great challenge for classification. In this paper, we retrieve knowledge from external knowledge source to enhance the semantic representation of short texts. We take conceptual information as a kind of knowledge and incorporate it into deep neural networks. For the purpose of measuring the importance of knowledge, we introduce attention mechanisms and propose deep Short Text Classification with Knowledge powered Attention (STCKA). We utilize Concept towards Short Text (CST) attention and Concept towards Concept Set (C-CS) attention to acquire the weight of concepts from two aspects. And we classify a short text with the help of conceptual information. Unlike traditional approaches, our model acts like a human being who has intrinsic ability to make decisions based on observation (i.e., training data for machines) and pays more attention to important knowledge. We also conduct extensive experiments on four public datasets for different tasks. The experimental results and case studies show that our model outperforms the state-of-the-art methods, justifying the effectiveness of knowledge powered attention.

Cognitive Aspects-Based Short Text Representation with Named Entity, Concept and Knowledge

Applied Sciences ◽

10.3390/app10144893 ◽

2020 ◽

Vol 10 (14) ◽

pp. 4893 ◽

Cited By ~ 1

Author(s):

Wenfeng Hou ◽

Qing Liu ◽

Longbing Cao

Keyword(s):

Text Classification ◽

Semantic Representation ◽

Semantic Features ◽

Text Representation ◽

Short Text ◽

Individual Level ◽

Named Entity ◽

Concept Knowledge ◽

Representation Model ◽

Multi Level

Short text is widely seen in applications including Internet of Things (IoT). The appropriate representation and classification of short text could be severely disrupted by the sparsity and shortness of short text. One important solution is to enrich short text representation by involving cognitive aspects of text, including semantic concept, knowledge, and category. In this paper, we propose a named Entity-based Concept Knowledge-Aware (ECKA) representation model which incorporates semantic information into short text representation. ECKA is a multi-level short text semantic representation model, which extracts the semantic features from the word, entity, concept and knowledge levels by CNN, respectively. Since word, entity, concept and knowledge entity in the same short text have different cognitive informativeness for short text classification, attention networks are formed to capture these category-related attentive representations from the multi-level textual features, respectively. The final multi-level semantic representations are formed by concatenating all of these individual-level representations, which are used for text classification. Experiments on three tasks demonstrate our method significantly outperforms the state-of-the-art methods.

IDC: Quantitative Evaluation Benchmark of Interpretation Methods for Deep Text Classification Models

10.21203/rs.3.rs-960359/v1 ◽

2021 ◽

Author(s):

Mohammed Khaleel ◽

Lei Qi ◽

Wallapak Tavanapong ◽

Johnny Wong ◽

Adisak Sukul ◽

...

Keyword(s):

Quantitative Evaluation ◽

Language Processing ◽

Text Classification ◽

Deep Neural Networks ◽

Performance Metrics ◽

Ground Truth ◽

Decision Making Process ◽

Research Attention ◽

Prediction Confidence ◽

Insight Into

Abstract Recent advances in deep neural networks have achieved outstanding success in natural language processing. Due to the success and the black-box nature of the deep models, interpretation methods that provide insight into the decision-making process of the models have received an influx of research attention. However, there is no quantitative evaluation comparing interpretation methods for text classification other than observing classification accuracy or prediction confidence when important word grams are removed. This is due to the lack of interpretation ground truth. Manual labeling of a large interpretation ground truth is time-consuming. We propose IDC, a new benchmark for quantitative evaluation of I nterpretation methods for D eep text C lassification models. IDC consists of three methods that take existing text classification ground truth and generate three corresponding pseudo-interpretation ground truth datasets. We propose to use interpretation recall, interpretation precision, and Cohen’s kappa inter-agreement as performance metrics. We used the pseudo ground truth datasets and the metrics to evaluate six interpretation methods.

Comparison of pretraining models and strategies for health-related social media text classification

10.1101/2021.09.28.21264253 ◽

2021 ◽

Author(s):

Yuting Guo ◽

Yao Ge ◽

Yuan-Chi Yang ◽

Mohammed Ali Al-Garadi ◽

Abeed Sarker

Keyword(s):

Social Media ◽

Language Processing ◽

Text Classification ◽

High Performance ◽

Language Models ◽

Learning Performance ◽

Health Related ◽

Social Media Text ◽

Performance Results ◽

Better Than

Motivation Pretrained contextual language models proposed in the recent past have been reported to achieve state-of-the-art performances in many natural language processing (NLP) tasks. There is a need to benchmark such models for targeted NLP tasks, and to explore effective pretraining strategies to improve machine learning performance. Results In this work, we addressed the task of health-related social media text classification. We benchmarked five models-RoBERTa, BERTweet, TwitterBERT, BioClinical_BERT, and BioBERT on 22 tasks. We attempted to boost performance for the best models by comparing distinct pretraining strategies-domain-adaptive pretraining (DAPT), source-adaptive pretraining (SAPT), and topic-specific pretraining (TSPT). RoBERTa and BERTweet performed comparably in most tasks, and better than others. For pretraining strategies, SAPT performed better or comparable to the off-the-shelf models, and significantly outperformed DAPT. SAPT+TSPT showed consistently high performance, with statistically significant improvement in one task. Our findings demonstrate that RoBERTa and BERTweet are excellent off-the-shelf models for health-related social media text classification, and extended pretraining using SAPT and TSPT can further improve performance.

A Text Abstraction Summary Model Based on BERT Word Embedding and Reinforcement Learning

Applied Sciences ◽

10.3390/app9214701 ◽

2019 ◽

Vol 9 (21) ◽

pp. 4701 ◽

Cited By ~ 8

Author(s):

Qicai Wang ◽

Peiyu Liu ◽

Zhenfang Zhu ◽

Hongxia Yin ◽

Qiuyue Zhang ◽

...

Keyword(s):

Reinforcement Learning ◽

Language Processing ◽

Evaluation Method ◽

Ground Truth ◽

Text Summarization ◽

Word Embedding ◽

Text Representation ◽

Daily Mail ◽

Automatic Text Summarization ◽

Automatic Text

As a core task of natural language processing and information retrieval, automatic text summarization is widely applied in many fields. There are two existing methods for text summarization task at present: abstractive and extractive. On this basis we propose a novel hybrid model of extractive-abstractive to combine BERT (Bidirectional Encoder Representations from Transformers) word embedding with reinforcement learning. Firstly, we convert the human-written abstractive summaries to the ground truth labels. Secondly, we use BERT word embedding as text representation and pre-train two sub-models respectively. Finally, the extraction network and the abstraction network are bridged by reinforcement learning. To verify the performance of the model, we compare it with the current popular automatic text summary model on the CNN/Daily Mail dataset, and use the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metrics as the evaluation method. Extensive experimental results show that the accuracy of the model is improved obviously.

An Ensemble Learning Strategy for Eligibility Criteria Text Classification for Clinical Trial Recruitment: Algorithm Development and Validation

JMIR Medical Informatics ◽

10.2196/17832 ◽

2020 ◽

Vol 8 (7) ◽

pp. e17832

Author(s):

Kun Zeng ◽

Zhiwei Pan ◽

Yibin Xu ◽

Yingying Qu

Keyword(s):

Clinical Trial ◽

Clinical Trials ◽

Natural Language Processing ◽

Ensemble Learning ◽

Language Processing ◽

Text Classification ◽

State Of The Art ◽

Shared Task ◽

Eligibility Criteria ◽

Short Text

Background Eligibility criteria are the main strategy for screening appropriate participants for clinical trials. Automatic analysis of clinical trial eligibility criteria by digital screening, leveraging natural language processing techniques, can improve recruitment efficiency and reduce the costs involved in promoting clinical research. Objective We aimed to create a natural language processing model to automatically classify clinical trial eligibility criteria. Methods We proposed a classifier for short text eligibility criteria based on ensemble learning, where a set of pretrained models was integrated. The pretrained models included state-of-the-art deep learning methods for training and classification, including Bidirectional Encoder Representations from Transformers (BERT), XLNet, and A Robustly Optimized BERT Pretraining Approach (RoBERTa). The classification results by the integrated models were combined as new features for training a Light Gradient Boosting Machine (LightGBM) model for eligibility criteria classification. Results Our proposed method obtained an accuracy of 0.846, a precision of 0.803, and a recall of 0.817 on a standard data set from a shared task of an international conference. The macro F1 value was 0.807, outperforming the state-of-the-art baseline methods on the shared task. Conclusions We designed a model for screening short text classification criteria for clinical trials based on multimodel ensemble learning. Through experiments, we concluded that performance was improved significantly with a model ensemble compared to a single model. The introduction of focal loss could reduce the impact of class imbalance to achieve better performance.

TextNAS: A Neural Architecture Search Space Tailored for Text Representation

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i05.6462 ◽

2020 ◽

Vol 34 (05) ◽

pp. 9242-9249

Author(s):

Yujing Wang ◽

Yaming Yang ◽

Yiren Chen ◽

Jing Bai ◽

Ce Zhang ◽

...

Keyword(s):

Text Classification ◽

Network Architecture ◽

State Of The Art ◽

Search Space ◽

Search Algorithms ◽

Text Representation ◽

Neural Architecture ◽

Good Potential ◽

Automatic Search ◽

Public Datasets

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.

SALTClass: classifying clinical short notes using background knowledge from unlabeled data

10.1101/801944 ◽

2019 ◽

Author(s):

Ayoub Bagheri ◽

Daniel Oberski ◽

Arjan Sammani ◽

Peter G.M. van der Heijden ◽

Folkert W. Asselbergs

Keyword(s):

Machine Learning ◽

Language Processing ◽

Text Classification ◽

Latent Dirichlet Allocation ◽

Machine Learning Algorithms ◽

Unlabeled Data ◽

Specific Information ◽

Short Text ◽

Link Type ◽

Python Package

AbstractBackgroundWith the increasing use of unstructured text in electronic health records, extracting useful related information has become a necessity. Text classification can be applied to extract patients’ medical history from clinical notes. However, the sparsity in clinical short notes, that is, excessively small word counts in the text, can lead to large classification errors. Previous studies demonstrated that natural language processing (NLP) can be useful in the text classification of clinical outcomes. We propose incorporating the knowledge from unlabeled data, as this may alleviate the problem of short noisy sparse text.ResultsThe software package SALTClass (short and long text classifier) is a machine learning NLP toolkit. It uses seven clustering algorithms, namely, latent Dirichlet allocation, K-Means, MiniBatchK-Means, BIRCH, MeanShift, DBScan, and GMM. Smoothing methods are applied to the resulting cluster information to enrich the representation of sparse text. For the subsequent prediction step, SALTClass can be used on either the original document-term matrix or in an enrichment pipeline. To this end, ten different supervised classifiers have also been integrated into SALTClass. We demonstrate the effectiveness of the SALTClass NLP toolkit in the identification of patients’ family history in a Dutch clinical cardiovascular text corpus from University Medical Center Utrecht, the Netherlands.ConclusionsThe considerable amount of unstructured short text in healthcare applications, particularly in clinical cardiovascular notes, has created an urgent need for tools that can parse specific information from text reports. Using machine learning algorithms for enriching short text can improve the representation for further applications.AvailabilitySALTClass can be downloaded as a Python package from Python Package Index (PyPI) website athttps://pypi.org/project/saltclassand from GitHub athttps://github.com/bagheria/saltclass.

Correlation-Guided Representation for Multi-Label Text Classification

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2021/463 ◽

2021 ◽

Author(s):

Qian-Wen Zhang ◽

Ximing Zhang ◽

Zhao Yan ◽

Ruifang Liu ◽

Yunbo Cao ◽

...

Keyword(s):

Language Processing ◽

Text Classification ◽

Low Frequency ◽

Classification Performance ◽

Categorical Variables ◽

Text Representation ◽

Label Semantics ◽

Higher Weights ◽

Label Correlations ◽

Text Information

Multi-label text classification is an essential task in natural language processing. Existing multi-label classification models generally consider labels as categorical variables and ignore the exploitation of label semantics. In this paper, we view the task as a correlation-guided text representation problem: an attention-based two-step framework is proposed to integrate text information and label semantics by jointly learning words and labels in the same space. In this way, we aim to capture high-order label-label correlations as well as context-label correlations. Specifically, the proposed approach works by learning token-level representations of words and labels globally through a multi-layer Transformer and constructing an attention vector through word-label correlation matrix to generate the text representation. It ensures that relevant words receive higher weights than irrelevant words and thus directly optimizes the classification performance. Extensive experiments over benchmark multi-label datasets clearly validate the effectiveness of the proposed approach, and further analysis demonstrates that it is competitive in both predicting low-frequency labels and convergence speed.

A Method of Short Text Representation Based on the Feature Probability Embedded Vector

Sensors ◽

10.3390/s19173728 ◽

2019 ◽

Vol 19 (17) ◽

pp. 3728 ◽

Cited By ~ 5

Author(s):

Zhou ◽

Wang ◽

Sun ◽

Sun

Keyword(s):

Language Processing ◽

Text Categorization ◽

Topic Model ◽

Main Idea ◽

Feature Weighting ◽

Word Embedding ◽

Text Representation ◽

Short Text ◽

Representation Method ◽

Feature Probability

Text representation is one of the key tasks in the field of natural language processing (NLP). Traditional feature extraction and weighting methods often use the bag-of-words (BoW) model, which may lead to a lack of semantic information as well as the problems of high dimensionality and high sparsity. At present, to solve these problems, a popular idea is to utilize deep learning methods. In this paper, feature weighting, word embedding, and topic models are combined to propose an unsupervised text representation method named the feature, probability, and word embedding method. The main idea is to use the word embedding technology Word2Vec to obtain the word vector, and then combine this with the feature weighted TF-IDF and the topic model LDA. Compared with traditional feature engineering, the proposed method not only increases the expressive ability of the vector space model, but also reduces the dimensions of the document vector. Besides this, it can be used to solve the problems of the insufficient information, high dimensions, and high sparsity of BoW. We use the proposed method for the task of text categorization and verify the validity of the method.