scholarly journals Study of Statistical Text Representation Methods for Performance Improvement of a Hierarchical Attention Network

2021 ◽  
Vol 11 (13) ◽  
pp. 6113
Author(s):  
Adam Wawrzyński ◽  
Julian Szymański
Keyword(s):  
Neural Networks ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Document Classification ◽  
Bag Of Words ◽  
Text Representation ◽  
Attention Network ◽  
Document Frequency ◽  
Textual Data ◽  
Latent Representations

To effectively process textual data, many approaches have been proposed to create text representations. The transformation of a text into a form of numbers that can be computed using computers is crucial for further applications in downstream tasks such as document classification, document summarization, and so forth. In our work, we study the quality of text representations using statistical methods and compare them to approaches based on neural networks. We describe in detail nine different algorithms used for text representation and then we evaluate five diverse datasets: BBCSport, BBC, Ohsumed, 20Newsgroups, and Reuters. The selected statistical models include Bag of Words (BoW), Term Frequency-Inverse Document Frequency (TFIDF) weighting, Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA). For the second group of deep neural networks, Partition-Smooth Inverse Frequency (P-SIF), Doc2Vec-Distributed Bag of Words Paragraph Vector (Doc2Vec-DBoW), Doc2Vec-Memory Model of Paragraph Vectors (Doc2Vec-DM), Hierarchical Attention Network (HAN) and Longformer were selected. The text representation methods were benchmarked in the document classification task and BoW and TFIDF models were used were used as a baseline. Based on the identified weaknesses of the HAN method, an improvement in the form of a Hierarchical Weighted Attention Network (HWAN) was proposed. The incorporation of statistical features into HAN latent representations improves or provides comparable results on four out of five datasets. The article presents how the length of the processed text affects the results of HAN and variants of HWAN models.

scholarly journals Topic Modelling Brazilian Supreme Court Lawsuits

2020 ◽  
Author(s):  
Pedro Henrique Luz De Araujo ◽  
Teófilo De Campos
Keyword(s):  
Supreme Court ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Classification Task ◽  
Bag Of Words ◽  
Topic Modelling ◽  
Text Representation ◽  
Good Classification ◽  
Low Dimensional ◽  
Topic Relevancy

The present work proposes the use of Latent Dirichlet Allocation to model Extraordinary Appeals received by Brazil’s Supreme Court. The data consist of a corpus of 45,532 lawsuits manually annotated by the Court’s experts with theme labels, a multi-class and multi-label classification task. We initially train models with 10 and 30 topics and analyze their semantics by examining each topic’s most relevant words and their most representative texts, aiming to evaluate model interpretability and quality. We also train models with 30, 100, 300 and 1,000 topics, and quantitatively evaluate their potential using the topics to generate feature vectors for each appeal. These vectors are then used to train a lawsuit theme classifier. We compare traditional bag-of-words approaches (word counts and tf-idf values) with the topic-based text representation to assess topic relevancy. Our topics semantic analysis demonstrate that our models with 10 and 30 topics were capable of capturing some of the legal matters discussed by the Court. In addition, our experiments show that the model with 300 topics was the best text vectoriser and that the interpretable, low dimensional representations it generates achieve good classification results.

scholarly journals Graph-based Representation for Sentence Similarity Measure : A Comparative Analysis

2018 ◽  
Vol 7 (2.14) ◽  
pp. 32
Author(s):  
Siti Sakira Kamaruddin ◽  
Yuhanis Yusof ◽  
Nur Azzah Abu Bakar ◽  
Mohamed Ahmed Tayie ◽  
Ghaith Abdulsattar A.Jabbar Alkubaisi
Keyword(s):  
Comparative Analysis ◽  
Similarity Measure ◽  
Semantic Analysis ◽  
Similarity Measures ◽  
Jaccard Similarity ◽  
Inverse Document Frequency ◽  
Document Frequency ◽  
Sentence Level ◽  
Textual Data ◽  
Document Level

Textual data are a rich source of knowledge; hence, sentence comparison has become one of the important tasks in text mining related works. Most previous work in text comparison are performed at document level, research suggest that comparing sentence level text is a non-trivial problem.  One of the reason is two sentences can convey the same meaning with totally dissimilar words.  This paper presents the results of a comparative analysis on three representation schemes i.e. term frequency inverse document frequency, Latent Semantic Analysis and Graph based representation using three similarity measures i.e. Cosine, Dice coefficient and Jaccard similarity to compare the similarity of sentences.  Results reveal that the graph based representation and the Jaccard similarity measure outperforms the others in terms of precision, recall and F-measures. 

scholarly journals Deep mixtures of unigrams for uncovering topics in textual data

2021 ◽  
Vol 31 (3) ◽  
Author(s):  
Cinzia Viroli ◽  
Laura Anderlucci
Keyword(s):  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Real Data ◽  
Unsupervised Classification ◽  
Adjusted Rand Index ◽  
Classification Rate ◽  
Affinity Propagation Clustering ◽  
The Em Algorithm ◽  
Textual Data ◽  
Cosine Distance

AbstractMixtures of unigrams are one of the simplest and most efficient tools for clustering textual data, as they assume that documents related to the same topic have similar distributions of terms, naturally described by multinomials. When the classification task is particularly challenging, such as when the document-term matrix is high-dimensional and extremely sparse, a more composite representation can provide better insight into the grouping structure. In this work, we developed a deep version of mixtures of unigrams for the unsupervised classification of very short documents with a large number of terms, by allowing for models with further deeper latent layers; the proposal is derived in a Bayesian framework. The behavior of the deep mixtures of unigrams is empirically compared with that of other traditional and state-of-the-art methods, namely k-means with cosine distance, k-means with Euclidean distance on data transformed according to semantic analysis, partition around medoids, mixture of Gaussians on semantic-based transformed data, hierarchical clustering according to Ward’s method with cosine dissimilarity, latent Dirichlet allocation, mixtures of unigrams estimated via the EM algorithm, spectral clustering and affinity propagation clustering. The performance is evaluated in terms of both correct classification rate and Adjusted Rand Index. Simulation studies and real data analysis prove that going deep in clustering such data highly improves the classification accuracy.

scholarly journals Semantic Similarity Analysis for Corpus Development and Paraphrase Detection in Arabic

2020 ◽  
Vol 18 (1) ◽  
pp. 1-7
Author(s):  
Adnen Mahmoud ◽  
Mounir Zrigui
Keyword(s):  
Language Processing ◽  
Latent Dirichlet Allocation ◽  
Question Answering ◽  
Semantic Analysis ◽  
Arabic Language ◽  
Plagiarism Detection ◽  
Part Of Speech ◽  
Document Frequency ◽  
Proposed Model ◽  
Semantic Similarity Analysis

Paraphrase detection allows determining how original and suspect documents convey the same meaning. It has attracted attention from researchers in many Natural Language Processing (NLP) tasks such as plagiarism detection, question answering, information retrieval, etc., Traditional methods (e.g., Term Frequency-Inverse Document Frequency (TF-IDF), Latent Dirichlet Allocation (LDA), and Latent Semantic Analysis (LSA)) cannot capture efficiently hidden semantic relations when sentences may not contain any common words or the co-occurrence of words is rarely present. Therefore, we proposed a deep learning model based on Global Word embedding (GloVe) and Recurrent Convolutional Neural Network (RCNN). It was efficient for capturing more contextual dependencies between words vectors with precise semantic meanings. Seeing the lack of resources in Arabic language publicly available, we developed a paraphrased corpus automatically. It preserved syntactic and semantic structures of Arabic sentences using word2vec model and Part-Of-Speech (POS) annotation. Overall experiments shown that our proposed model outperformed the state-of-the-art methods in terms of precision and recall

scholarly journals The Performance of BERT as Data Representation of Text Clustering

2021 ◽  
Author(s):  
Alvin Subakti ◽  
Hendri Murfi ◽  
Nora Hariadi
Keyword(s):  
Feature Extraction ◽  
Clustering Algorithm ◽  
Clustering Algorithms ◽  
Text Clustering ◽  
Data Representation ◽  
Text Representation ◽  
Inverse Document Frequency ◽  
Document Frequency ◽  
Normalization Methods ◽  
Textual Data

Abstract Text clustering is the task of grouping a set of texts so that text in the same group will be more similar than those from a different group. The process of grouping text manually requires a significant amount of time and labor. Therefore, automation utilizing machine learning is necessary. The standard method used to represent textual data is Term Frequency Inverse Document Frequency (TFIDF). However, TFIDF cannot consider the position and context of a word in a sentence. Bidirectional Encoder Representation from Transformers (BERT) model can produce text representation that incorporates the position and context of a word in a sentence. This research analyzed the performance of the BERT model as data representation for text. Moreover, various feature extraction and normalization methods are also applied for the data representation of the BERT model. To examine the performances of BERT, we use four clustering algorithms, i.e., k-means clustering, eigenspace-based fuzzy c-means, deep embedded clustering, and improved deep embedded clustering. Our simulations show that BERT outperforms the standard TFIDF method in 28 out of 36 metrics. Furthermore, different feature extraction and normalization produced varied performances. The usage of these feature extraction and normalization must be altered depending on the text clustering algorithm used.

A Comparison on the Use of LSA and LDA in Psychology Analysis on “Courage” Definitions

2017 ◽  
Vol 11 (03) ◽  
pp. 373-389
Author(s):  
Sara Santilli ◽  
Laura Nota ◽  
Giovanni Pilato
Keyword(s):  
Latent Semantic Analysis ◽  
Computational Models ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Data Sets ◽  
The Public ◽  
Textual Data ◽  
Definition Of ◽  
Italian Context ◽  
Compare And Contrast

In the present work Latent Semantic Analysis of textual data was applied on texts related to courage, in order to compare and contrast results and evaluate the opportunity of integrating different data sets. To better understand the definition of courage in Italian context, 1199 participants were involved in the present study and was asked to answer to the following question “Courage is[Formula: see text]”. The participants’ definitions of courage were analyzed with the Latent Semantic Analysis (LSA), and Latent Dirichlet Allocation (LDA), in order to study the fundamental concepts arising from the population. An analogous comparison with Twitter posts has been also carried out to analyze if the public opinion emerging from social media provides a challenging and rich context to explore computational models of natural language.

scholarly journals Root cause analysis of COVID-19 cases by enhanced text mining process

2022 ◽  
Vol 12 (2) ◽  
pp. 1807
Author(s):  
Sujatha Arun Kokatnoor ◽  
Balachandran Krishnan
Keyword(s):  
Dirichlet Process ◽  
Clustering Algorithm ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Learning Approaches ◽  
The Public ◽  
Root Cause ◽  
Document Frequency ◽  
Coherence Score ◽  
Index Value

<p>The main focus of this research is to find the reasons behind the fresh cases of COVID-19 from the public’s perception for data specific to India. The analysis is done using machine learning approaches and validating the inferences with medical professionals. The data processing and analysis is accomplished in three steps. First, the dimensionality of the vector space model (VSM) is reduced with improvised feature engineering (FE) process by using a weighted term frequency-inverse document frequency (TF-IDF) and forward scan trigrams (FST) followed by removal of weak features using feature hashing technique. In the second step, an enhanced K-means clustering algorithm is used for grouping, based on the public posts from Twitter®. In the last step, latent dirichlet allocation (LDA) is applied for discovering the trigram topics relevant to the reasons behind the increase of fresh COVID-19 cases. The enhanced K-means clustering improved Dunn index value by 18.11% when compared with the traditional K-means method. By incorporating improvised two-step FE process, LDA model improved by 14% in terms of coherence score and by 19% and 15% when compared with latent semantic analysis (LSA) and hierarchical dirichlet process (HDP) respectively thereby resulting in 14 root causes for spike in the disease.</p>

Similarity Detection Using Latent Semantic Analysis Algorithm

2018 ◽  
Vol 6 (8) ◽  
pp. 102
Author(s):  
Priyanka R. Patil ◽  
Shital A. Patil
Keyword(s):  
Latent Semantic Analysis ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Mining Method ◽  
Research Papers ◽  
Information Measures ◽  
Automated Software ◽  
Day By Day ◽  
Ways Of Life ◽  
Dirichlet Allocation

Similarity View is an application for visually comparing and exploring multiple models of text and collection of document. Friendbook finds ways of life of clients from client driven sensor information, measures the closeness of ways of life amongst clients, and prescribes companions to clients if their ways of life have high likeness. Roused by demonstrate a clients day by day life as life records, from their ways of life are separated by utilizing the Latent Dirichlet Allocation Algorithm. Manual techniques can't be utilized for checking research papers, as the doled out commentator may have lacking learning in the exploration disciplines. For different subjective views, causing possible misinterpretations. An urgent need for an effective and feasible approach to check the submitted research papers with support of automated software. A method like text mining method come to solve the problem of automatically checking the research papers semantically. The proposed method to finding the proper similarity of text from the collection of documents by using Latent Dirichlet Allocation (LDA) algorithm and Latent Semantic Analysis (LSA) with synonym algorithm which is used to find synonyms of text index wise by using the English wordnet dictionary, another algorithm is LSA without synonym used to find the similarity of text based on index. LSA with synonym rate of accuracy is greater when the synonym are consider for matching.

Designing a Chat-Bot for College Information using Information Retrieval and Automatic Text Summarization Techniques

2020 ◽  
Vol 01 ◽  
Author(s):  
Radha Guha
Keyword(s):  
Information Retrieval ◽  
Language Processing ◽  
Latent Dirichlet Allocation ◽  
Semantic Analysis ◽  
Text Summarization ◽  
The Internet ◽  
Specific Domain ◽  
User Query ◽  
College Information ◽  
Chat Bot

Background:: In the era of information overload it is very difficult for a human reader to make sense of the vast information available in the internet quickly. Even for a specific domain like college or university website it may be difficult for a user to browse through all the links to get the relevant answers quickly. Objective:: In this scenario, design of a chat-bot which can answer questions related to college information and compare between colleges will be very useful and novel. Methods:: In this paper a novel conversational interface chat-bot application with information retrieval and text summariza-tion skill is designed and implemented. Firstly this chat-bot has a simple dialog skill when it can understand the user query intent, it responds from the stored collection of answers. Secondly for unknown queries, this chat-bot can search the internet and then perform text summarization using advanced techniques of natural language processing (NLP) and text mining (TM). Results:: The advancement of NLP capability of information retrieval and text summarization using machine learning tech-niques of Latent Semantic Analysis(LSI), Latent Dirichlet Allocation (LDA), Word2Vec, Global Vector (GloVe) and Tex-tRank are reviewed and compared in this paper first before implementing them for the chat-bot design. This chat-bot im-proves user experience tremendously by getting answers to specific queries concisely which takes less time than to read the entire document. Students, parents and faculty can get the answers for variety of information like admission criteria, fees, course offerings, notice board, attendance, grades, placements, faculty profile, research papers and patents etc. more effi-ciently. Conclusion:: The purpose of this paper was to follow the advancement in NLP technologies and implement them in a novel application.

Sign in / Sign up

Export Citation Format

Share Document