scholarly journals Dataless Short Text Classification Based on Biterm Topic Model and Word Embeddings

2020 ◽  
Author(s):  
Yi Yang ◽  
Hongan Wang ◽  
Jiaqi Zhu ◽  
Yunkun Wu ◽  
Kailong Jiang ◽  
...  
Keyword(s):  
Text Classification ◽  
Classification Accuracy ◽  
Topic Model ◽  
State Of The Art ◽  
The Internet ◽  
Word Embeddings ◽  
Word Similarity ◽  
Short Text ◽  
Category Word ◽  
Novel Model

Dataless text classification has attracted increasing attentions recently. It only needs very few seed words of each category to classify documents, which is much cheaper than supervised text classification that requires massive labeling efforts. However, most of existing models pay attention to long texts, but get unsatisfactory performance on short texts, which have become increasingly popular on the Internet. In this paper, we at first propose a novel model named Seeded Biterm Topic Model (SeedBTM) extending BTM to solve the problem of dataless short text classification with seed words. It takes advantage of both word co-occurrence information in the topic model and category-word similarity from widely used word embeddings as the prior topic-in-set knowledge. Moreover, with the same approach, we also propose Seeded Twitter Biterm Topic Model (SeedTBTM), which extends Twitter-BTM and utilizes additional user information to achieve higher classification accuracy. Experimental results on five real short-text datasets show that our models outperform the state-of-the-art methods, and especially perform well when the categories are overlapping and interrelated.

scholarly journals A Topical Word Embeddings for Text Classification

2018 ◽  
Author(s):  
João Marcos Carvalho Lima ◽  
José Everardo Bessa Maia
Keyword(s):  
Text Classification ◽  
Text Categorization ◽  
Topic Model ◽  
State Of The Art ◽  
Bag Of Words ◽  
Word Embeddings ◽  
Document Representation ◽  
Linear Classifier ◽  
Low Dimensionality ◽  
Nonlinear Classifier

This paper presents an approach that uses topic models based on LDA to represent documents in text categorization problems. The document representation is achieved through the cosine similarity between document embeddings and embeddings of topic words, creating a Bag-of-Topics (BoT) variant. The performance of this approach is compared against those of two other representations: BoW (Bag-of-Words) and Topic Model, both based on standard tf-idf. Also, to reveal the effect of the classifier, we compared the performance of the nonlinear classifier SVM against that of the linear classifier Naive Bayes, taken as baseline. To evaluate the approach we use two bases, one multi-label (RCV-1) and another single-label (20 Newsgroup). The model presents significant results with low dimensionality when compared to the state of the art.

Incorporate Syntactic Information for Short Text Classification

2011 ◽  
Vol 268-270 ◽  
pp. 697-700
Author(s):  
Rui Xue Duan ◽  
Xiao Jie Wang ◽  
Wen Feng Li
Keyword(s):  
Machine Learning ◽  
Feature Selection ◽  
Learning Environment ◽  
Text Classification ◽  
The Internet ◽  
Selection Methods ◽  
Text Documents ◽  
Short Text ◽  
Syntactic Information ◽  
Dependency Relations

As the volume of online short text documents grow tremendously on the Internet, it is much more urgent to solve the task of organizing the short texts well. However, the traditional feature selection methods cannot suitable for the short text. In this paper, we proposed a method to incorporate syntactic information for the short text. It emphasizes the feature which has more dependency relations with other words. The classifier SVM and machine learning environment Weka are involved in our experiments. The experiment results show that incorporate syntactic information in the short text, we can get more powerful features than traditional feature selection methods, such as DF, CHI. The precision of short text classification improved from 86.2% to 90.8%.

scholarly journals Filtering and Classifying Relevant Short Text with a Few Seed Words

2019 ◽  
Vol 3 (3) ◽  
pp. 165-186 ◽  
Author(s):  
Chenliang Li ◽  
Shiqian Chen ◽  
Yan Qi
Keyword(s):  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
State Of The Art ◽  
Superior Performance ◽  
Support Vector ◽  
Short Text ◽  
Text Filtering ◽  
Supervised Classifiers ◽  
Real World Datasets ◽  
Weakly Supervised

Abstract Filtering out irrelevant documents and classifying the relevant ones into topical categories is a de facto task in many applications. However, supervised learning solutions require extravagant human efforts on document labeling. In this paper, we propose a novel seed-guided topic model for dataless short text classification and filtering, named SSCF. Without using any labeled documents, SSCF takes a few “seed words” for each category of interest, and conducts short text filtering and classification in a weakly supervised manner. To overcome the issues of data sparsity and imbalance, the short text collection is mapped to a collection of pseudodocuments, one for each word. SSCF infers two kinds of topics on pseudo-documents: category-topics and general-topics. Each category-topic is associated with one category of interest, covering the meaning of the latter. In SSCF, we devise a novel word relevance estimation process based on the seed words, for hidden topic inference. The dominating topic of a short text is identified through post inference and then used for filtering and classification. On two real-world datasets in two languages, experimental results show that our proposed SSCF consistently achieves better classification accuracy than state-of-the-art baselines. We also observe that SSCF can even achieve superior performance than the supervised classifiers supervised latent dirichlet allocation (sLDA) and support vector machine (SVM) on some testing tasks.

scholarly journals A New Vector Representation of Short Texts for Classification

2019 ◽  
Vol 17 (2) ◽  
pp. 241-249
Author(s):  
Yangyang Li ◽  
Bo Liu
Keyword(s):  
Text Classification ◽  
Web Search ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Classification Performance ◽  
New Method ◽  
Data Sets ◽  
Text Data ◽  
Short Text ◽  
Space Model

Short and sparse characteristics and synonyms and homonyms are main obstacles for short-text classification. In recent years, research on short-text classification has focused on expanding short texts but has barely guaranteed the validity of expanded words. This study proposes a new method to weaken these effects without external knowledge. The proposed method analyses short texts by using the topic model based on Latent Dirichlet Allocation (LDA), represents each short text by using a vector space model and presents a new method to adjust the vector of short texts. In the experiments, two open short-text data sets composed of google news and web search snippets are utilised to evaluate the classification performance and prove the effectiveness of our method.

scholarly journals A Sparse Topic Model for Bursty Topic Discovery in Social Networks

2020 ◽  
Vol 17 (5) ◽  
pp. 816-824
Author(s):  
Lei Shi ◽  
Junping Du ◽  
Feifei Kou
Keyword(s):  
Topic Model ◽  
State Of The Art ◽  
Sina Weibo ◽  
Short Text ◽  
Qualitative And Quantitative ◽  
Topic Discovery ◽  
Spike And Slab Prior ◽  
Automatic Discovery ◽  
The Common ◽  
Sparse Topic Model

Bursty topic discovery aims to automatically identify bursty events and continuously keep track of known events. The existing methods focus on the topic model. However, the sparsity of short text brings the challenge to the traditional topic models because the words are too few to learn from the original corpus. To tackle this problem, we propose a Sparse Topic Model (STM) for bursty topic discovery. First, we distinguish the modeling between the bursty topic and the common topic to detect the change of the words in time and discover the bursty words. Second, we introduce “Spike and Slab” prior to decouple the sparsity and smoothness of a distribution. The bursty words are leveraged to achieve automatic discovery of the bursty topics. Finally, to evaluate the effectiveness of our proposed algorithm, we collect Sina weibo dataset to conduct various experiments. Both qualitative and quantitative evaluations demonstrate that the proposed STM algorithm outperforms favorably against several state-of-the-art methods

SenU-PTM: a novel phrase-based topic model for short-text topic discovery by exploiting word embeddings

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Heng-Yang Lu ◽  
Yi Zhang ◽  
Yuntao Du
Keyword(s):  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Second Phase ◽  
Word Embeddings ◽  
Two Phase ◽  
Content Type ◽  
Short Text ◽  
Topic Discovery ◽  
Two Phases ◽  
Sense Unit

PurposeTopic model has been widely applied to discover important information from a vast amount of unstructured data. Traditional long-text topic models such as Latent Dirichlet Allocation may suffer from the sparsity problem when dealing with short texts, which mostly come from the Web. These models also exist the readability problem when displaying the discovered topics. The purpose of this paper is to propose a novel model called the Sense Unit based Phrase Topic Model (SenU-PTM) for both the sparsity and readability problems.Design/methodology/approachSenU-PTM is a novel phrase-based short-text topic model under a two-phase framework. The first phase introduces a phrase-generation algorithm by exploiting word embeddings, which aims to generate phrases with the original corpus. The second phase introduces a new concept of sense unit, which consists of a set of semantically similar tokens for modeling topics with token vectors generated in the first phase. Finally, SenU-PTM infers topics based on the above two phases.FindingsExperimental results on two real-world and publicly available datasets show the effectiveness of SenU-PTM from the perspectives of topical quality and document characterization. It reveals that modeling topics on sense units can solve the sparsity of short texts and improve the readability of topics at the same time.Originality/valueThe originality of SenU-PTM lies in the new procedure of modeling topics on the proposed sense units with word embeddings for short-text topic discovery.

Using Word Embeddings with Linear Models for Short Text Classification

2018 ◽  
pp. 819-827
Author(s):  
Alfred Krzywicki ◽  
Bradford Heap ◽  
Michael Bain ◽  
Wayne Wobcke ◽  
Susanne Schmeidl
Keyword(s):  
Text Classification ◽  
Linear Models ◽  
Word Embeddings ◽  
Short Text

scholarly journals Pronunciation-Enhanced Chinese Word Embedding

2021 ◽  
Author(s):  
Qinjuan Yang ◽  
Haoran Xie ◽  
Gary Cheng ◽  
Fu Lee Wang ◽  
Yanghui Rao
Keyword(s):  
Sentiment Analysis ◽  
Text Classification ◽  
Semantic Information ◽  
Word Embedding ◽  
Chinese Characters ◽  
Learning Method ◽  
Word Embeddings ◽  
Chinese Word ◽  
Word Similarity ◽  
Meaning Structure

AbstractChinese word embeddings have recently garnered considerable attention. Chinese characters and their sub-character components, which contain rich semantic information, are incorporated to learn Chinese word embeddings. Chinese characters can represent a combination of meaning, structure, and pronunciation. However, existing embedding learning methods focus on the structure and meaning of Chinese characters. In this study, we aim to develop an embedding learning method that can make complete use of the information represented by Chinese characters, including phonology, morphology, and semantics. Specifically, we propose a pronunciation-enhanced Chinese word embedding learning method, where the pronunciations of context characters and target characters are simultaneously encoded into the embeddings. Evaluation of word similarity, word analogy reasoning, text classification, and sentiment analysis validate the effectiveness of our proposed method.

scholarly journals An Ensemble Learning Strategy for Eligibility Criteria Text Classification for Clinical Trial Recruitment: Algorithm Development and Validation

10.2196/17832 ◽  
2020 ◽  
Vol 8 (7) ◽  
pp. e17832
Author(s):  
Kun Zeng ◽  
Zhiwei Pan ◽  
Yibin Xu ◽  
Yingying Qu
Keyword(s):  
Clinical Trial ◽  
Clinical Trials ◽  
Natural Language Processing ◽  
Ensemble Learning ◽  
Language Processing ◽  
Text Classification ◽  
State Of The Art ◽  
Shared Task ◽  
Eligibility Criteria ◽  
Short Text

Background Eligibility criteria are the main strategy for screening appropriate participants for clinical trials. Automatic analysis of clinical trial eligibility criteria by digital screening, leveraging natural language processing techniques, can improve recruitment efficiency and reduce the costs involved in promoting clinical research. Objective We aimed to create a natural language processing model to automatically classify clinical trial eligibility criteria. Methods We proposed a classifier for short text eligibility criteria based on ensemble learning, where a set of pretrained models was integrated. The pretrained models included state-of-the-art deep learning methods for training and classification, including Bidirectional Encoder Representations from Transformers (BERT), XLNet, and A Robustly Optimized BERT Pretraining Approach (RoBERTa). The classification results by the integrated models were combined as new features for training a Light Gradient Boosting Machine (LightGBM) model for eligibility criteria classification. Results Our proposed method obtained an accuracy of 0.846, a precision of 0.803, and a recall of 0.817 on a standard data set from a shared task of an international conference. The macro F1 value was 0.807, outperforming the state-of-the-art baseline methods on the shared task. Conclusions We designed a model for screening short text classification criteria for clinical trials based on multimodel ensemble learning. Through experiments, we concluded that performance was improved significantly with a model ensemble compared to a single model. The introduction of focal loss could reduce the impact of class imbalance to achieve better performance.

Sign in / Sign up

Export Citation Format

Share Document