Can topic models be used in research evaluations? Reproducibility, validity, and reliability when compared with semantic maps

2019 ◽  
Vol 28 (3) ◽  
pp. 263-272 ◽  
Author(s):  
Tobias Hecking ◽  
Loet Leydesdorff
Keyword(s):  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Research Evaluation ◽  
Topic Models ◽  
Principal Component ◽  
Evaluation Framework ◽  
Validity And Reliability ◽  
Text Corpora ◽  
Semantic Maps ◽  
Semantic Coherence

AbstractWe replicate and analyze the topic model which was commissioned to King’s College and Digital Science for the Research Evaluation Framework (REF 2014) in the United Kingdom: 6,638 case descriptions of societal impact were submitted by 154 higher-education institutes. We compare the Latent Dirichlet Allocation (LDA) model with Principal Component Analysis (PCA) of document-term matrices using the same data. Since topic models are almost by definition applied to text corpora which are too large to read, validation of the results of these models is hardly possible; furthermore the models are irreproducible for a number of reasons. However, removing a small fraction of the documents from the sample—a test for reliability—has on average a larger impact in terms of decay on LDA than on PCA-based models. The semantic coherence of LDA models outperforms PCA-based models. In our opinion, results of the topic models are statistical and should not be used for grant selections and micro decision-making about research without follow-up using domain-specific semantic maps.

scholarly journals Topic Modeling in Embedding Spaces

2020 ◽  
Vol 8 ◽  
pp. 439-453 ◽  
Author(s):  
Adji B. Dieng ◽  
Francisco J. R. Ruiz ◽  
David M. Blei
Keyword(s):  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Topic Models ◽  
Predictive Performance ◽  
Inner Product ◽  
Natural Parameter ◽  
Document Models ◽  
Heavy Tailed ◽  
Categorical Distribution

Topic modeling analyzes documents to learn meaningful patterns of words. However, existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. To this end, we develop the embedded topic model (etm), a generative model of documents that marries traditional topic models with word embeddings. More specifically, the etm models each word with a categorical distribution whose natural parameter is the inner product between the word’s embedding and an embedding of its assigned topic. To fit the etm, we develop an efficient amortized variational inference algorithm. The etm discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms existing document models, such as latent Dirichlet allocation, in terms of both topic quality and predictive performance.

Ldagibbs: A Command for Topic Modeling in Stata Using Latent Dirichlet Allocation

2018 ◽  
Vol 18 (1) ◽  
pp. 101-117 ◽  
Author(s):  
Carlo Schwarz
Keyword(s):  
Machine Learning ◽  
Probability Distribution ◽  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Topic Models ◽  
Text Documents ◽  
Text Data ◽  
Dirichlet Allocation

In this article, I introduce the ldagibbs command, which implements latent Dirichlet allocation in Stata. Latent Dirichlet allocation is the most popular machine-learning topic model. Topic models automatically cluster text documents into a user-chosen number of topics. Latent Dirichlet allocation represents each document as a probability distribution over topics and represents each topic as a probability distribution over words. Therefore, latent Dirichlet allocation provides a way to analyze the content of large unclassified text data and an alternative to predefined document classifications.

scholarly journals Method of Moments for Topic Models with Mixed Discrete and Continuous Features

2021 ◽  
Author(s):  
Joachim Giesen ◽  
Paul Kahlmeyer ◽  
Sören Laue ◽  
Matthias Mitterreiter ◽  
Frank Nussbaum ◽  
...  
Keyword(s):  
Method Of Moments ◽  
Latent Variable ◽  
Latent Dirichlet Allocation ◽  
Latent Class ◽  
Topic Model ◽  
Likelihood Function ◽  
Natural Extension ◽  
Topic Models ◽  
Discrete Variables ◽  
Continuous State

Topic models are characterized by a latent class variable that represents the different topics. Traditionally, their observable variables are modeled as discrete variables like, for instance, in the prototypical latent Dirichlet allocation (LDA) topic model. In LDA, words in text documents are encoded by discrete count vectors with respect to some dictionary. The classical approach for learning topic models optimizes a likelihood function that is non-concave due to the presence of the latent variable. Hence, this approach mostly boils down to using search heuristics like the EM algorithm for parameter estimation. Recently, it was shown that topic models can be learned with strong algorithmic and statistical guarantees through Pearson's method of moments. Here, we extend this line of work to topic models that feature discrete as well as continuous observable variables (features). Moving beyond discrete variables as in LDA allows for more sophisticated features and a natural extension of topic models to other modalities than text, like, for instance, images. We provide algorithmic and statistical guarantees for the method of moments applied to the extended topic model that we corroborate experimentally on synthetic data. We also demonstrate the applicability of our model on real-world document data with embedded images that we preprocess into continuous state-of-the-art feature vectors.

ERD-MedLDA: Entity relation detection using supervised topic models with maximum margin learning

2012 ◽  
Vol 18 (2) ◽  
pp. 263-289 ◽  
Author(s):  
DINGCHENG LI ◽  
SWAPNA SOMASUNDARAN ◽  
AMIT CHAKRABORTY
Keyword(s):  
Topic Modeling ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Topic Models ◽  
Likelihood Estimation ◽  
Named Entities ◽  
Maximum Margin ◽  
Heterogeneous Features ◽  
Overall Performance ◽  
Entity Relation Detection

AbstractThis paper proposes a novel application of topic models to do entity relation detection (ERD). In order to make use of the latent semantics of text, we formulate the task of relation detection as a topic modeling problem. The motivation is to find underlying topics that are indicative of relations between named entities (NEs). Our approach considers pairs of NEs and features associated with them as mini documents, and aims to utilize the underlying topic distributions as indicators for the types of relations that may exist between the NE pair. Our system, ERD-MedLDA, adapts Maximum Entropy Discriminant Latent Dirichlet Allocation (MedLDA) with mixed membership for relation detection. By using supervision, ERD-MedLDA is able to learn topic distributions indicative of relation types. Further, ERD-MedLDA is a topic model that combines the benefits of both, maximum likelihood estimation (MLE) and maximum margin estimation (MME), and the mixed-membership formulation enables the system to incorporate heterogeneous features. We incorporate different features into the system and perform experiments on the ACE 2005 corpus. Our approach achieves better overall performance for precision, recall, and F-measure metrics as compared to baseline SVM-based and LDA-based models. We also find that our system shows better and consistent improvements with the addition of complex informative features as compared to baseline systems.

scholarly journals No Longer Lost in Translation: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications

2017 ◽  
Author(s):  
Erik de Vries ◽  
Martijn Schoonvelde ◽  
Gijs Schumacher
Keyword(s):  
Machine Translation ◽  
Text Analysis ◽  
Gold Standard ◽  
Topic Model ◽  
Topic Models ◽  
Bag Of Words ◽  
Text Corpora ◽  
Automated Text Analysis ◽  
Lost In Translation

Automated text analysis allows researchers to analyze large quantities of text. Yet, comparative researchers are presented with a big challenge: across countries people speak different languages. To address this issue, some analysts have suggested using Google Translate to convert all texts into English before starting the analysis (Lucas et al., 2015). But in doing so, do we get lost in translation? This paper evaluates the usefulness of machine translation for bag-of-words models – such as topic models. We use the europarl dataset and compare term-document matrices as well as topic model results from gold standard translated text and machine-translated text. We evaluate results at both the document and the corpus level. We first find term-document matrices for both text corpora to be highly similar, with significant but minor differences across languages. What is more, we find considerable overlap in the set of features generated from human-translated and machine-translated texts. With regards to LDA topic models, we find topical prevalence and topical content to be highly similar with only small differences across languages. We conclude that Google Translate is a useful tool for comparative researchers when using bag-of-words text models.

scholarly journals Detection of Cases of Noncompliance to Drug Treatment in Patient Forum Posts: Topic Model Approach (Preprint)

2017 ◽  
Author(s):  
Redhouane Abdellaoui ◽  
Pierre Foulquié ◽  
Nathalie Texier ◽  
Carole Faviez ◽  
Anita Burgun ◽  
...  
Keyword(s):  
Social Media ◽  
Virtual Communities ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Antidepressant Drug ◽  
Topic Models ◽  
Web Crawler ◽  
Probabilistic Topic Model ◽  
Manual Review ◽  
Model Approach

BACKGROUND Medication nonadherence is a major impediment to the management of many health conditions. A better understanding of the factors underlying noncompliance to treatment may help health professionals to address it. Patients use peer-to-peer virtual communities and social media to share their experiences regarding their treatments and diseases. Using topic models makes it possible to model themes present in a collection of posts, thus to identify cases of noncompliance. OBJECTIVE The aim of this study was to detect messages describing patients’ noncompliant behaviors associated with a drug of interest. Thus, the objective was the clustering of posts featuring a homogeneous vocabulary related to nonadherent attitudes. METHODS We focused on escitalopram and aripiprazole used to treat depression and psychotic conditions, respectively. We implemented a probabilistic topic model to identify the topics that occurred in a corpus of messages mentioning these drugs, posted from 2004 to 2013 on three of the most popular French forums. Data were collected using a Web crawler designed by Kappa Santé as part of the Detec’t project to analyze social media for drug safety. Several topics were related to noncompliance to treatment. RESULTS Starting from a corpus of 3650 posts related to an antidepressant drug (escitalopram) and 2164 posts related to an antipsychotic drug (aripiprazole), the use of latent Dirichlet allocation allowed us to model several themes, including interruptions of treatment and changes in dosage. The topic model approach detected cases of noncompliance behaviors with a recall of 98.5% (272/276) and a precision of 32.6% (272/844). CONCLUSIONS Topic models enabled us to explore patients’ discussions on community websites and to identify posts related with noncompliant behaviors. After a manual review of the messages in the noncompliance topics, we found that noncompliance to treatment was present in 6.17% (276/4469) of the posts.

scholarly journals Latent Feature Word Representations to Enhance Topic Models for Text Mining Algorithms

2019 ◽  
Vol 9 (2) ◽  
pp. 4816-4821
Keyword(s):  
Text Mining ◽  
Process Model ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Document Clustering ◽  
Topic Models ◽  
Mining Operations ◽  
Document Collections ◽  
Generative Process ◽  
Document Collection

Dealing with large number of textual documents needs proven models that leverage the efficiency in processing. Text mining needs such models to have meaningful approaches to extract latent features from document collection. Latent Dirichlet allocation (LDA) is one such probabilistic generative process model that helps in representing document collections in a systematic approach. In many text mining applications LDA is useful as it supports many models. One such model is known as Topic Model. However, topic models LDA needs to be improved in order to exploit latent feature vector representations of words trained on large corpora to improve word-topic mapping learnt on smaller corpus. With respect to document clustering and document classification, it is essential to have a novel topic models to improve performance. In this paper, an improved topic model is proposed and implemented using LDA which exploits the benefits of Word2Vec tool to have pre-trained word vectors so as to achieve the desired enhancement. A prototype application is built to demonstrate the proof of the concept with text mining operations like document clustering.

Trends in Transportation Research

2017 ◽  
Vol 2614 (1) ◽  
pp. 27-38 ◽  
Author(s):  
Subasish Das ◽  
Karen Dixon ◽  
Xiaoduan Sun ◽  
Anandi Dutta ◽  
Michelle Zupancich
Keyword(s):  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
Topic Models ◽  
Machine Learning Algorithms ◽  
Main Theme ◽  
Transportation Engineering ◽  
Engineering Research ◽  
Textual Data ◽  
Transportation Research ◽  
Conference Papers

Proceedings of journal and conference papers are good sources of big textual data to examine research trends in various branches of science. The contents, usually unstructured in nature, require fast machine-learning algorithms to be deciphered. Exploratory analysis through text mining usually provides the descriptive nature of the contents but lacks quantification of the topics and their correlations. Topic models are algorithms designed to discover the main theme or trend in massive collections of unstructured documents. Through the use of a structural topic model, an extension of latent Dirichlet allocation, this study introduced distinct topic models on the basis of the relative frequencies of the words used in the abstracts of 15,357 TRB compendium papers. With data from 7 years (2008 through 2014) of TRB annual meeting compendium papers, the 20 most dominant topics emerged from a bag of 4 million words. The findings of this study contributed to the understanding of topical trends in the complex and evolving field of transportation engineering research.

scholarly journals No Longer Lost in Translation: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications

2018 ◽  
Vol 26 (4) ◽  
pp. 417-430 ◽  
Author(s):  
Erik de Vries ◽  
Martijn Schoonvelde ◽  
Gijs Schumacher
Keyword(s):  
Machine Translation ◽  
Text Analysis ◽  
Gold Standard ◽  
Topic Model ◽  
Topic Models ◽  
Bag Of Words ◽  
Text Corpora ◽  
Automated Text Analysis ◽  
Lost In Translation

Automated text analysis allows researchers to analyze large quantities of text. Yet, comparative researchers are presented with a big challenge: across countries people speak different languages. To address this issue, some analysts have suggested using Google Translate to convert all texts into English before starting the analysis (Lucas et al. 2015). But in doing so, do we get lost in translation? This paper evaluates the usefulness of machine translation for bag-of-words models—such as topic models. We use the europarl dataset and compare term-document matrices (TDMs) as well as topic model results from gold standard translated text and machine-translated text. We evaluate results at both the document and the corpus level. We first find TDMs for both text corpora to be highly similar, with minor differences across languages. What is more, we find considerable overlap in the set of features generated from human-translated and machine-translated texts. With regard to LDA topic models, we find topical prevalence and topical content to be highly similar with again only small differences across languages. We conclude that Google Translate is a useful tool for comparative researchers when using bag-of-words text models.

scholarly journals Topic-sensitive multi-document summarization algorithm

2015 ◽  
Vol 12 (4) ◽  
pp. 1375-1389
Author(s):  
Liu Na ◽  
Tang Di ◽  
Lu Ying ◽  
Tang Xiao-Jun ◽  
Wang Hai-Wen
Keyword(s):  
Linear Combination ◽  
Latent Dirichlet Allocation ◽  
Topic Model ◽  
State Of The Art ◽  
Sentence Length ◽  
Combination Strategy ◽  
Document Summarization ◽  
Text Corpora ◽  
Equal Importance ◽  
Weight Calculation

Latent Dirichlet Allocation (LDA) has been used to generate text corpora topics recently. However, not all the estimated topics are of equal importance or correspond to genuine themes of the domain. Some of the topics can be a collection of irrelevant words or represent insignificant themes. This paper proposed a topic-sensitive algorithm for multi-document summarization. This algorithm uses LDA model and weight linear combination strategy to identify significance topic which is used in sentence weight calculation. Each topic is measured by three different LDA criteria. Significance topic is evaluated by using weight linear combination to combine the multi-criteria. In addition to topic features, the proposed approach also considered some statistics features, such as term frequency, sentence position, sentence length, etc. It not only highlights the advantages of statistics features, but also cooperates with topic model. The experiments showed that the proposed algorithm achieves better performance than the other state-of-the-art algorithms on DUC2002 corpus.

Sign in / Sign up

Export Citation Format

Share Document