scholarly journals It all starts with entities: A Salient entity topic model

2019 ◽  
Vol 26 (5) ◽  
pp. 531-549
Author(s):  
Chuan Wu ◽  
Evangelos Kanoulas ◽  
Maarten de Rijke
Keyword(s):  
Topic Model ◽  
State Of The Art ◽  
Topic Models ◽  
Generation Process ◽  
Qualitative And Quantitative Analysis ◽  
Generative Process ◽  
Qualitative And Quantitative ◽  
Proposed Model ◽  
Topic Distribution ◽  
Document Generation

AbstractEntities play an essential role in understanding textual documents, regardless of whether the documents are short, such as tweets, or long, such as news articles. In short textual documents, all entities mentioned are usually considered equally important because of the limited amount of information. In long textual documents, however, not all entities are equally important: some are salient and others are not. Traditional entity topic models (ETMs) focus on ways to incorporate entity information into topic models to better explain the generative process of documents. However, entities are usually treated equally, without considering whether they are salient or not. In this work, we propose a novel ETM, Salient Entity Topic Model, to take salient entities into consideration in the document generation process. In particular, we model salient entities as a source of topics used to generate words in documents, in addition to the topic distribution of documents used in traditional topic models. Qualitative and quantitative analysis is performed on the proposed model. Application to entity salience detection demonstrates the effectiveness of our model compared to the state-of-the-art topic model baselines.

scholarly journals Document Informed Neural Autoregressive Topic Models with Distributional Prior

2019 ◽  
Vol 33 ◽  
pp. 6505-6512 ◽  
Author(s):  
Pankaj Gupta ◽  
Yatin Chaudhary ◽  
Florian Buettner ◽  
Hinrich Schütze
Keyword(s):  
Topic Model ◽  
State Of The Art ◽  
Topic Models ◽  
Language Modeling ◽  
Context Information ◽  
Short Text ◽  
Neuron Networks ◽  
Proposed Model ◽  
Actual Meaning ◽  
Biological Neuron

We address two challenges in topic models: (1) Context information around words helps in determining their actual meaning, e.g., “networks” used in the contexts artificial neural networks vs. biological neuron networks. Generative topic models infer topic-word distributions, taking no or only little context into account. Here, we extend a neural autoregressive topic model to exploit the full context information around words in a document in a language modeling fashion. The proposed model is named as iDocNADE. (2) Due to the small number of word occurrences (i.e., lack of context) in short text and data sparsity in a corpus of few documents, the application of topic models is challenging on such texts. Therefore, we propose a simple and efficient way of incorporating external knowledge into neural autoregressive topic models: we use embeddings as a distributional prior. The proposed variants are named as DocNADEe and iDocNADEe. We present novel neural autoregressive topic model variants that consistently outperform state-of-the-art generative topic models in terms of generalization, interpretability (topic coherence) and applicability (retrieval and classification) over 7 long-text and 8 short-text datasets from diverse domains.

Author–Subject–Topic model for reviewer recommendation

2018 ◽  
Vol 45 (4) ◽  
pp. 554-570 ◽  
Author(s):  
Jian Jin ◽  
Qian Geng ◽  
Haikun Mou ◽  
Chong Chen
Keyword(s):  
Information System ◽  
Topic Model ◽  
Academic Library ◽  
Topic Models ◽  
Interdisciplinary Studies ◽  
Distribution Analysis ◽  
Topic Distribution ◽  
Research Domains

Interdisciplinary studies are becoming increasingly popular, and research domains of many experts are becoming diverse. This phenomenon brings difficulty in recommending experts to review interdisciplinary submissions. In this study, an Author–Subject–Topic (AST) model is proposed with two versions. In the model, reviewers’ subject information is embedded to analyse topic distributions of submissions and reviewers’ publications. The major difference between the AST and Author–Topic models lies in the introduction of a ‘Subject’ layer, which supervises the generation of hierarchical topics and allows sharing of subjects among authors. To evaluate the performance of the AST model, papers in Information System and Management (a typical interdisciplinary domain) in a famous Chinese academic library are investigated. Comparative experiments are conducted, which show the effectiveness of the AST model in topic distribution analysis and reviewer recommendation for interdisciplinary studies.

scholarly journals Modeling Topical Relevance for Multi-Turn Dialogue Generation

2020 ◽  
Author(s):  
Hainan Zhang ◽  
Yanyan Lan ◽  
Liang Pang ◽  
Hongshen Chen ◽  
Zhuoye Ding ◽  
...  
Keyword(s):  
Topic Model ◽  
State Of The Art ◽  
Training Dataset ◽  
Common Phenomenon ◽  
New Model ◽  
Sentence Level ◽  
Topic Distribution ◽  
Topic Drift ◽  
Relevant Context ◽  
Customer Services

Topic drift is a common phenomenon in multi-turn dialogue. Therefore, an ideal dialogue generation models should be able to capture the topic information of each context, detect the relevant context, and produce appropriate responses accordingly. However, existing models usually use word or sentence level similarities to detect the relevant contexts, which fail to well capture the topical level relevance. In this paper, we propose a new model, named STAR-BTM, to tackle this problem. Firstly, the Biterm Topic Model is pre-trained on the whole training dataset. Then, the topic level attention weights are computed based on the topic representation of each context. Finally, the attention weights and the topic distribution are utilized in the decoding process to generate the corresponding responses. Experimental results on both Chinese customer services data and English Ubuntu dialogue data show that STAR-BTM significantly outperforms several state-of-the-art methods, in terms of both metric-based and human evaluations.

scholarly journals A Method for Constructing Supervised Topic Model Based on Term Frequency-Inverse Topic Frequency

Symmetry ◽  
2019 ◽  
Vol 11 (12) ◽  
pp. 1486
Author(s):  
Zhinan Gou ◽  
Zheng Huo ◽  
Yuanzhen Liu ◽  
Yi Yang
Keyword(s):  
Topic Modeling ◽  
Topic Model ◽  
State Of The Art ◽  
Topic Models ◽  
Document Classification ◽  
Experimental Results ◽  
Tag Recommendation ◽  
Term Frequency ◽  
Series Of Experiments ◽  
Dirichlet Prior

Supervised topic modeling has been successfully applied in the fields of document classification and tag recommendation in recent years. However, most existing models neglect the fact that topic terms have the ability to distinguish topics. In this paper, we propose a term frequency-inverse topic frequency (TF-ITF) method for constructing a supervised topic model, in which the weight of each topic term indicates the ability to distinguish topics. We conduct a series of experiments with not only the symmetric Dirichlet prior parameters but also the asymmetric Dirichlet prior parameters. Experimental results demonstrate that the result of introducing TF-ITF into a supervised topic model outperforms several state-of-the-art supervised topic models.

scholarly journals EXTRACTING TOPICS FROM A TV CHANNEL'S FACEBOOK PAGE USING CONTEXTUALIZED DOCUMENT EMBEDDING

2021 ◽  
Vol XLVI-4/W5-2021 ◽  
pp. 245-249
Author(s):  
N. Habbat ◽  
H. Anoun ◽  
L. Hassouni
Keyword(s):  
Neural Network ◽  
Topic Model ◽  
Topic Models ◽  
Neural Model ◽  
Contextual Knowledge ◽  
Development Level ◽  
Proposed Model ◽  
Transformer Model

Abstract. Topic models extract meaningful words from text collection, allowing for a better understanding of data. However, the results are often not coherent enough, and thus harder to interpret. Adding more contextual knowledge to the model can enhance coherence. In recent years, neural network-based topic models become available, and the development level of the neural model has developed thanks to BERT-based representation. In this study, we suggest a model extract news on the Aljazeera Facebook page. Our approach combines the neural model (ProdLDA) and the Arabic Pre-training BERT transformer model (AraBERT). Therefore, the proposed model produces more expressive and consistent topics than ELMO using different topic model algorithms (ProdLDA and LDA) with 0.883 in topic coherence.

scholarly journals A Sparse Topic Model for Bursty Topic Discovery in Social Networks

2020 ◽  
Vol 17 (5) ◽  
pp. 816-824
Author(s):  
Lei Shi ◽  
Junping Du ◽  
Feifei Kou
Keyword(s):  
Topic Model ◽  
State Of The Art ◽  
Sina Weibo ◽  
Short Text ◽  
Qualitative And Quantitative ◽  
Topic Discovery ◽  
Spike And Slab Prior ◽  
Automatic Discovery ◽  
The Common ◽  
Sparse Topic Model

Bursty topic discovery aims to automatically identify bursty events and continuously keep track of known events. The existing methods focus on the topic model. However, the sparsity of short text brings the challenge to the traditional topic models because the words are too few to learn from the original corpus. To tackle this problem, we propose a Sparse Topic Model (STM) for bursty topic discovery. First, we distinguish the modeling between the bursty topic and the common topic to detect the change of the words in time and discover the bursty words. Second, we introduce “Spike and Slab” prior to decouple the sparsity and smoothness of a distribution. The bursty words are leveraged to achieve automatic discovery of the bursty topics. Finally, to evaluate the effectiveness of our proposed algorithm, we collect Sina weibo dataset to conduct various experiments. Both qualitative and quantitative evaluations demonstrate that the proposed STM algorithm outperforms favorably against several state-of-the-art methods

scholarly journals A Method for Constructing Supervised Time Topic Model Based on Variational Autoencoder

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Zhinan Gou ◽  
Yan Li ◽  
Zheng Huo
Keyword(s):  
Topic Modeling ◽  
Topic Model ◽  
State Of The Art ◽  
Generation Model ◽  
Construction Process ◽  
Research Papers ◽  
Generative Process ◽  
Model Based ◽  
Variational Autoencoder ◽  
Over Time

Topic modeling is a probabilistic generation model to find the representative topic of a document and has been successfully applied to various document-related tasks in recent years. Especially in the supervised topic model and time topic model, many methods have achieved some success. The supervised topic model can learn topics from documents annotated with multiple labels and the time topic model can learn topics that evolve over time in a sequentially organized corpus. However, there are some documents with multiple labels and time-stamped in reality, which need to construct a supervised time topic model to achieve document-related tasks. There are few research papers on the supervised time topic model. To solve this problem, we propose a method for constructing a supervised time topic model. By analysing the generative process of the supervised topic model and time topic model, respectively, we introduce the construction process of the supervised time topic model based on variational autoencoder in detail and conduct preliminary experiments. Experimental results demonstrate that the supervised time topic model outperforms several state-of-the-art topic models.

scholarly journals 3-in-1 Correlated Embedding via Adaptive Exploration of the Structure and Semantic Subspaces

2018 ◽  
Author(s):  
Liang Yang ◽  
Yuanfang Guo ◽  
Di Jin ◽  
Huazhu Fu ◽  
Xiaochun Cao
Keyword(s):  
Real World ◽  
Posterior Distribution ◽  
State Of The Art ◽  
Variational Inference ◽  
Experimental Results ◽  
Generation Process ◽  
Network Embedding ◽  
Topological Information ◽  
Proposed Model ◽  
Types Of Information

Combinational  network embedding, which learns the node representation by exploring both  topological and non-topological information, becomes popular due to the fact that the two types of information are complementing each other.  Most of the existing methods either consider the  topological and non-topological  information being aligned or possess predetermined preferences during the embedding process.Unfortunately, previous methods  fail to either explicitly describe the correlations between topological and non-topological information or adaptively weight their impacts. To address the existing issues, three new assumptions are proposed to better describe the embedding space and its properties. With the proposed assumptions, nodes, communities and topics are mapped into one embedding space. A novel generative model is proposed to formulate the generation process of the network and content from the embeddings, with respect to the Bayesian framework. The proposed model automatically leans to the information which is more discriminative.The embedding result can be obtained by maximizing the posterior distribution by adopting the variational inference and reparameterization trick. Experimental results indicate that the proposed method gives superior performances compared to the state-of-the-art methods when a variety of real-world networks is analyzed.

scholarly journals Mining heterogeneous clinical notes by multi-modal latent topic model

PLoS ONE ◽  
2021 ◽  
Vol 16 (4) ◽  
pp. e0249622
Author(s):  
Zhi Wen ◽  
Pratheeksha Nair ◽  
Chih-Ying Deng ◽  
Xing Han Lu ◽  
Edward Moseley ◽  
...  
Keyword(s):  
Mechanical Ventilation ◽  
Topic Model ◽  
Prolonged Mechanical Ventilation ◽  
Topic Models ◽  
Clinical Notes ◽  
Topic Distribution ◽  
Domain Expertise ◽  
Latent Topic ◽  
Poor Outcomes ◽  
Mimic Iii

Latent knowledge can be extracted from the electronic notes that are recorded during patient encounters with the health system. Using these clinical notes to decipher a patient’s underlying comorbidites, symptom burdens, and treatment courses is an ongoing challenge. Latent topic model as an efficient Bayesian method can be used to model each patient’s clinical notes as “documents” and the words in the notes as “tokens”. However, standard latent topic models assume that all of the notes follow the same topic distribution, regardless of the type of note or the domain expertise of the author (such as doctors or nurses). We propose a novel application of latent topic modeling, using multi-note topic model (MNTM) to jointly infer distinct topic distributions of notes of different types. We applied our model to clinical notes from the MIMIC-III dataset to infer distinct topic distributions over the physician and nursing note types. Based on manual assessments made by clinicians, we observed a significant improvement in topic interpretability using MNTM modeling over the baseline single-note topic models that ignore the note types. Moreover, our MNTM model led to a significantly higher prediction accuracy for prolonged mechanical ventilation and mortality using only the first 48 hours of patient data. By correlating the patients’ topic mixture with hospital mortality and prolonged mechanical ventilation, we identified several diagnostic topics that are associated with poor outcomes. Because of its elegant and intuitive formation, we envision a broad application of our approach in mining multi-modality text-based healthcare information that goes beyond clinical notes. Code available at https://github.com/li-lab-mcgill/heterogeneous_ehr.

scholarly journals Latent Feature Word Representations to Enhance Topic Models for Text Mining Algorithms

2019 ◽  
Vol 9 (2) ◽  
pp. 4816-4821
Keyword(s):  
Text Mining ◽  
Process Model ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Document Clustering ◽  
Topic Models ◽  
Mining Operations ◽  
Document Collections ◽  
Generative Process ◽  
Document Collection

Dealing with large number of textual documents needs proven models that leverage the efficiency in processing. Text mining needs such models to have meaningful approaches to extract latent features from document collection. Latent Dirichlet allocation (LDA) is one such probabilistic generative process model that helps in representing document collections in a systematic approach. In many text mining applications LDA is useful as it supports many models. One such model is known as Topic Model. However, topic models LDA needs to be improved in order to exploit latent feature vector representations of words trained on large corpora to improve word-topic mapping learnt on smaller corpus. With respect to document clustering and document classification, it is essential to have a novel topic models to improve performance. In this paper, an improved topic model is proposed and implemented using LDA which exploits the benefits of Word2Vec tool to have pre-trained word vectors so as to achieve the desired enhancement. A prototype application is built to demonstrate the proof of the concept with text mining operations like document clustering.

Sign in / Sign up

Export Citation Format

Share Document