scholarly journals Estimating Topic Modeling Performance with Sharma–Mittal Entropy

Entropy ◽  
2019 ◽  
Vol 21 (7) ◽  
pp. 660 ◽  
Author(s):  
Sergei Koltcov ◽  
Vera Ignatenko ◽  
Olessia Koltsova
Keyword(s):  
Latent Semantic Analysis ◽  
Topic Modeling ◽  
Statistical Physics ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Probabilistic Latent Semantic Analysis ◽  
Model Parameters ◽  
Theoretical Ground ◽  
Text Documents ◽  
Optimizing Model

Topic modeling is a popular approach for clustering text documents. However, current tools have a number of unsolved problems such as instability and a lack of criteria for selecting the values of model parameters. In this work, we propose a method to solve partially the problems of optimizing model parameters, simultaneously accounting for semantic stability. Our method is inspired by the concepts from statistical physics and is based on Sharma–Mittal entropy. We test our approach on two models: probabilistic Latent Semantic Analysis (pLSA) and Latent Dirichlet Allocation (LDA) with Gibbs sampling, and on two datasets in different languages. We compare our approach against a number of standard metrics, each of which is able to account for just one of the parameters of our interest. We demonstrate that Sharma–Mittal entropy is a convenient tool for selecting both the number of topics and the values of hyper-parameters, simultaneously controlling for semantic stability, which none of the existing metrics can do. Furthermore, we show that concepts from statistical physics can be used to contribute to theory construction for machine learning, a rapidly-developing sphere that currently lacks a consistent theoretical ground.

scholarly journals Analyzing the Influence of Hyper-parameters and Regularizers of Topic Modeling in Terms of Renyi Entropy

Entropy ◽  
2020 ◽  
Vol 22 (4) ◽  
pp. 394
Author(s):  
Sergei Koltcov ◽  
Vera Ignatenko ◽  
Zeyd Boukhers ◽  
Steffen Staab
Keyword(s):  
Gibbs Sampling ◽  
Dirichlet Process ◽  
Topic Modeling ◽  
Statistical Physics ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Topic Models ◽  
Renyi Entropy ◽  
Rényi Entropy ◽  
Probabilistic Latent Semantic Analysis

Topic modeling is a popular technique for clustering large collections of text documents. A variety of different types of regularization is implemented in topic modeling. In this paper, we propose a novel approach for analyzing the influence of different regularization types on results of topic modeling. Based on Renyi entropy, this approach is inspired by the concepts from statistical physics, where an inferred topical structure of a collection can be considered an information statistical system residing in a non-equilibrium state. By testing our approach on four models—Probabilistic Latent Semantic Analysis (pLSA), Additive Regularization of Topic Models (BigARTM), Latent Dirichlet Allocation (LDA) with Gibbs sampling, LDA with variational inference (VLDA)—we, first of all, show that the minimum of Renyi entropy coincides with the “true” number of topics, as determined in two labelled collections. Simultaneously, we find that Hierarchical Dirichlet Process (HDP) model as a well-known approach for topic number optimization fails to detect such optimum. Next, we demonstrate that large values of the regularization coefficient in BigARTM significantly shift the minimum of entropy from the topic number optimum, which effect is not observed for hyper-parameters in LDA with Gibbs sampling. We conclude that regularization may introduce unpredictable distortions into topic models that need further research.

scholarly journals STABILITY OF TOPIC MODELING VIA MODALITY REGULARIZATION

2020 ◽  
Author(s):  
R. Derbanosov ◽  
◽  
M. Bakhanova ◽  
◽  
Keyword(s):  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Topic Model ◽  
Side Information ◽  
Auxiliary Information ◽  
Discrete Distributions ◽  
Probabilistic Latent Semantic Analysis ◽  
Probabilistic Topic Modeling ◽  
Random Initialization

Probabilistic topic modeling is a tool for statistical text analysis that can give us information about the inner structure of a large corpus of documents. The most popular models—Probabilistic Latent Semantic Analysis and Latent Dirichlet Allocation—produce topics in a form of discrete distributions over the set of all words of the corpus. They build topics using an iterative algorithm that starts from some random initialization and optimizes a loss function. One of the main problems of topic modeling is sensitivity to random initialization that means producing significantly different solutions from different initial points. Several studies showed that side information about documents may improve the overall quality of a topic model. In this paper, we consider the use of additional information in the context of the stability problem. We represent auxiliary information as an additional modality and use BigARTM library in order to perform experiments on several text collections. We show that using side information as an additional modality improves topics stability without significant quality loss of the model.

scholarly journals An Efficient Topic Modeling Approach for Text Mining and Information Retrieval through K-means Clustering

2020 ◽  
Vol 39 (1) ◽  
pp. 213-222
Author(s):  
Junaid Rashid ◽  
Syed Muhammad Adnan Shah ◽  
Aun Irtaza
Keyword(s):  
Information Retrieval ◽  
Text Mining ◽  
Topic Modeling ◽  
Clustering Algorithm ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
State Of The Art ◽  
Text Documents ◽  
New Perspective ◽  
Better Than

Topic modeling is an effective text mining and information retrieval approach to organizing knowledge with various contents under a specific topic. Text documents in form of news articles are increasing very fast on the web. Analysis of these documents is very important in the fields of text mining and information retrieval. Meaningful information extraction from these documents is a challenging task. One approach for discovering the theme from text documents is topic modeling but this approach still needs a new perspective to improve its performance. In topic modeling, documents have topics and topics are the collection of words. In this paper, we propose a new k-means topic modeling (KTM) approach by using the k-means clustering algorithm. KTM discovers better semantic topics from a collection of documents. Experiments on two real-world Reuters 21578 and BBC News datasets show that KTM performance is better than state-of-the-art topic models like LDA (Latent Dirichlet Allocation) and LSA (Latent Semantic Analysis). The KTM is also applicable for classification and clustering tasks in text mining and achieves higher performance with a comparison of its competitors LDA and LSA.

scholarly journals Topic Modeling as a Tool for Analyzing Library Chat Transcripts

2021 ◽  
Vol 40 (3) ◽  
Author(s):  
HyunSeung Koh ◽  
Mark Fienup
Keyword(s):  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Academic Library ◽  
Qualitative Evaluation ◽  
Probabilistic Latent Semantic Analysis ◽  
Analysis Tool ◽  
Chat Reference ◽  
Better Than

Library chat services are an increasingly important communication channel to connect patrons to library resources and services. Analysis of chat transcripts could provide librarians with insights into improving services. Unfortunately, chat transcripts consist of unstructured text data, making it impractical for librarians to go beyond simple quantitative analysis (e.g., chat duration, message count, word frequencies) with existing tools. As a stepping-stone toward a more sophisticated chat transcript analysis tool, this study investigated the application of different types of topic modeling techniques to analyze one academic library’s chat reference data collected from April 10, 2015, to May 31, 2019, with the goal of extracting the most accurate and easily interpretable topics. In this study, topic accuracy and interpretability—the quality of topic outcomes—were quantitatively measured with topic coherence metrics. Additionally, qualitative accuracy and interpretability were measured by the librarian author of this paper depending on the subjective judgment on whether topics are aligned with frequently asked questions or easily inferable themes in academic library contexts. This study found that from a human’s qualitative evaluation, Probabilistic Latent Semantic Analysis (pLSA) produced more accurate and interpretable topics, which is not necessarily aligned with the findings of the quantitative evaluation with all three types of topic coherence metrics. Interestingly, the commonly used technique Latent Dirichlet Allocation (LDA) did not necessarily perform better than pLSA. Also, semi-supervised techniques with human-curated anchor words of Correlation Explanation (CorEx) or guided LDA (GuidedLDA) did not necessarily perform better than an unsupervised technique of Dirichlet Multinomial Mixture (DMM). Last, the study found that using the entire transcript, including both sides of the interaction between the library patron and the librarian, performed better than using only the initial question asked by the library patron across different techniques in increasing the quality of topic outcomes.

Semantic Pattern Detection in Covid-19 using Contextual Clustering and Intelligent Topic Modeling

2022 ◽  
Vol 13 (2) ◽  
pp. 0-0
Keyword(s):  
Latent Semantic Analysis ◽  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Pattern Detection ◽  
Future Research ◽  
Detection Approach ◽  
Semantic Spaces ◽  
And Control ◽  
The Impact

The Covid-19 pandemic is the deadliest outbreak in our living memory. So, it is need of hour, to prepare the world with strategies to prevent and control the impact of the epidemics. In this paper, a novel semantic pattern detection approach in the Covid-19 literature using contextual clustering and intelligent topic modeling is presented. For contextual clustering, three level weights at term level, document level, and corpus level are used with latent semantic analysis. For intelligent topic modeling, semantic collocations using pointwise mutual information(PMI) and log frequency biased mutual dependency(LBMD) are selected and latent dirichlet allocation is applied. Contextual clustering with latent semantic analysis presents semantic spaces with high correlation in terms at corpus level. Through intelligent topic modeling, topics are improved in the form of lower perplexity and highly coherent. This research helps in finding the knowledge gap in the area of Covid-19 research and offered direction for future research.

Issues and Methods for Access, Storage, and Analysis of Data From Online Social Communities

2018 ◽  
pp. 402-432
Author(s):  
Christopher John Quinn ◽  
Matthew James Quinn ◽  
Alan Olinsky ◽  
John Thomas Quinn
Keyword(s):  
Social Network ◽  
Data Storage ◽  
Latent Semantic Analysis ◽  
Information Diffusion ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Online Social Network ◽  
Network Models ◽  
Probabilistic Latent Semantic Analysis ◽  
User Interactions

This chapter provides an overview for a number of important issues related to studying user interactions in an online social network. The approach of social network analysis is detailed along with important basic concepts for network models. The different ways of indicating influence within a network are provided by describing various measures such as degree centrality, betweenness centrality and closeness centrality. Network structure as represented by cliques and components with measures of connectedness defined by clustering and reciprocity are also included. With the large volume of data associated with social networks, the significance of data storage and sampling are discussed. Since verbal communication is significant within networks, textual analysis is reviewed with respect to classification techniques such as sentiment analysis and with respect to topic modeling specifically latent semantic analysis, probabilistic latent semantic analysis, latent Dirichlet allocation and alternatives. Another important area that is provided in detail is information diffusion.

scholarly journals Renormalization Analysis of Topic Models

Entropy ◽  
2020 ◽  
Vol 22 (5) ◽  
pp. 556
Author(s):  
Sergei Koltcov ◽  
Vera Ignatenko
Keyword(s):  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Optimal Number ◽  
Probabilistic Latent Semantic Analysis ◽  
Model Parameters ◽  
Grid Search ◽  
Renormalization Procedure ◽  
Allocation Model ◽  
Latent Dirichlet Allocation Model ◽  
Dirichlet Allocation

In practice, to build a machine learning model of big data, one needs to tune model parameters. The process of parameter tuning involves extremely time-consuming and computationally expensive grid search. However, the theory of statistical physics provides techniques allowing us to optimize this process. The paper shows that a function of the output of topic modeling demonstrates self-similar behavior under variation of the number of clusters. Such behavior allows using a renormalization technique. A combination of renormalization procedure with the Renyi entropy approach allows for quick searching of the optimal number of topics. In this paper, the renormalization procedure is developed for the probabilistic Latent Semantic Analysis (pLSA), and the Latent Dirichlet Allocation model with variational Expectation–Maximization algorithm (VLDA) and the Latent Dirichlet Allocation model with granulated Gibbs sampling procedure (GLDA). The experiments were conducted on two test datasets with a known number of topics in two different languages and on one unlabeled test dataset with an unknown number of topics. The paper shows that the renormalization procedure allows for finding an approximation of the optimal number of topics at least 30 times faster than the grid search without significant loss of quality.

scholarly journals Latent acoustic topic models for unstructured audio classification

2012 ◽  
Vol 1 ◽  
Author(s):  
Samuel Kim ◽  
Panayiotis Georgiou ◽  
Shrikanth Narayanan
Keyword(s):  
Latent Semantic Analysis ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Topic Models ◽  
Audio Signal ◽  
Gaussian Mixture ◽  
Probabilistic Latent Semantic Analysis ◽  
Support Vector ◽  
Audio Classification ◽  
Audio Clip

We propose the notion of latent acoustic topics to capture contextual information embedded within a collection of audio signals. The central idea is to learn a probability distribution over a set of latent topics of a given audio clip in an unsupervised manner, assuming that there exist latent acoustic topics and each audio clip can be described in terms of those latent acoustic topics. In this regard, we use the latent Dirichlet allocation (LDA) to implement the acoustic topic models over elemental acoustic units, referred as acoustic words, and perform text-like audio signal processing. Experiments on audio tag classification with the BBC sound effects library demonstrate the usefulness of the proposed latent audio context modeling schemes. In particular, the proposed method is shown to be superior to other latent structure analysis methods, such as latent semantic analysis and probabilistic latent semantic analysis. We also demonstrate that topic models can be used as complementary features to content-based features and offer about 9% relative improvement in audio classification when combined with the traditional Gaussian mixture model (GMM)–Support Vector Machine (SVM) technique.

scholarly journals Semantic Pattern Detection in COVID-19 Using Contextual Clustering and Intelligent Topic Modeling

2022 ◽  
Vol 13 (2) ◽  
pp. 1-17
Author(s):  
Pooja Kherwa ◽  
Poonam Bansal
Keyword(s):  
Latent Semantic Analysis ◽  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Pattern Detection ◽  
Future Research ◽  
Detection Approach ◽  
Semantic Spaces ◽  
And Control ◽  
The Impact

The Covid-19 pandemic is the deadliest outbreak in our living memory. So, it is need of hour, to prepare the world with strategies to prevent and control the impact of the epidemics. In this paper, a novel semantic pattern detection approach in the Covid-19 literature using contextual clustering and intelligent topic modeling is presented. For contextual clustering, three level weights at term level, document level, and corpus level are used with latent semantic analysis. For intelligent topic modeling, semantic collocations using pointwise mutual information(PMI) and log frequency biased mutual dependency(LBMD) are selected and latent dirichlet allocation is applied. Contextual clustering with latent semantic analysis presents semantic spaces with high correlation in terms at corpus level. Through intelligent topic modeling, topics are improved in the form of lower perplexity and highly coherent. This research helps in finding the knowledge gap in the area of Covid-19 research and offered direction for future research.

scholarly journals A comparative analysis of Latent Semantic analysis and Latent Dirichlet allocation topic modeling methods using Bible data

2020 ◽  
Vol 13 (44) ◽  
pp. 4474-4482
Author(s):  
Vasantha Kumari Garbhapu ◽  
Keyword(s):  
Latent Semantic Analysis ◽  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Word Association ◽  
Machine Learning Algorithms ◽  
Superior Performance ◽  
Data Set ◽  
Document Similarity ◽  
Dirichlet Allocation

Objective: To compare the topic modeling techniques, as no free lunch theorem states that under a uniform distribution over search problems, all machine learning algorithms perform equally. Hence, here, we compare Latent Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA) to identify better performer for English bible data set which has not been studied yet. Methods: This comparative study divided into three levels: In the first level, bible data was extracted from the sources and preprocessed to remove the words and characters which were not useful to obtain the semantic structures or necessary patterns to make the meaningful corpus. In the second level, the preprocessed data were converted into a bag of words and numerical statistic TF-IDF (Term Frequency – Inverse Document Frequency) is used to assess how relevant a word is to a document in a corpus. In the third level, Latent Semantic analysis and Latent Dirichlet Allocations methods were applied over the resultant corpus to study the feasibility of the techniques. Findings: Based on our evaluation, we observed that the LDA achieves 60 to 75% superior performance when compared to LSA using document similarity within-corpus, document similarity with the unseen document. Additionally, LDA showed better coherence score (0.58018) than LSA (0.50395). Moreover, when compared to any word within-corpus, the word association showed better results with LDA. Some words have homonyms based on the context; for example, in the bible; bear has a meaning of punishment and birth. In our study, LDA word association results are almost near to human word associations when compared to LSA. Novelty: LDA was found to be the computationally efficient and interpretable method in adopting the English Bible dataset of New International Version that was not yet created. Keywords: Topic modeling; LSA; LDA; word association; document similarity;Bible data set

Sign in / Sign up

Export Citation Format

Share Document