Dual Scaling in Data Mining from Text Databases

2006 ◽  
Vol 10 (4) ◽  
pp. 451-457 ◽  
Author(s):  
Junzo Watada ◽  
◽  
Keisuke Aoki ◽  
Masahiro Kawano ◽  
Muhammad Suzuri Hitam ◽  
...  
Keyword(s):  
Multivariate Analysis ◽  
Text Mining ◽  
Kansei Engineering ◽  
Semantic Meaning ◽  
Dual Scaling ◽  
Text Documents ◽  
Text Data ◽  
Text Document ◽  
Text Information ◽  
Quantification Model

The availability of multimedia text document information has disseminated text mining among researchers. Text documents, integrate numerical and linguistic data, making text mining interesting and challenging. We propose text mining based on a fuzzy quantification model and fuzzy thesaurus. In text mining, we focus on: 1) Sentences included in Japanese text that are broken down into words. 2) Fuzzy thesaurus for finding words matching keywords in text. 3) Fuzzy multivariate analysis to analyze semantic meaning in predefined case studies. We use a fuzzy thesaurus to translate words using Chinese and Japanese characters into keywords. This speeds up processing without requiring a dictionary to separate words. Fuzzy multivariate analysis is used to analyze such processed data and to extract latent mutual related structures in text data, i.e., to extract otherwise obscured knowledge. We apply dual scaling to mining library and Web page text information, and propose integrating the result in Kansei engineering for possible application in sales, marketing, and production.

An Improved B-hill Climbing Optimization Technique for Solving the Text Documents Clustering Problem

2020 ◽  
Vol 16 (4) ◽  
pp. 296-306 ◽  
Author(s):  
Laith Mohammad Abualigah ◽  
Essam Said Hanandeh ◽  
Ahamad Tajudin Khader ◽  
Mohammed Abdallh Otair ◽  
Shishir Kumar Shandilya
Keyword(s):  
Optimization Technique ◽  
Document Clustering ◽  
Text Clustering ◽  
Hill Climbing ◽  
Text Documents ◽  
Clustering Problem ◽  
Text Document ◽  
Text Information ◽  
Amount Of Knowledge ◽  
The Hill

Background: Considering the increasing volume of text document information on Internet pages, dealing with such a tremendous amount of knowledge becomes totally complex due to its large size. Text clustering is a common optimization problem used to manage a large amount of text information into a subset of comparable and coherent clusters. Aims: This paper presents a novel local clustering technique, namely, β-hill climbing, to solve the problem of the text document clustering through modeling the β-hill climbing technique for partitioning the similar documents into the same cluster. Methods: The β parameter is the primary innovation in β-hill climbing technique. It has been introduced in order to perform a balance between local and global search. Local search methods are successfully applied to solve the problem of the text document clustering such as; k-medoid and kmean techniques. Results: Experiments were conducted on eight benchmark standard text datasets with different characteristics taken from the Laboratory of Computational Intelligence (LABIC). The results proved that the proposed β-hill climbing achieved better results in comparison with the original hill climbing technique in solving the text clustering problem. Conclusion: The performance of the text clustering is useful by adding the β operator to the hill climbing.

Assessment of Twitter Data Clusters with Cosine-Based Validation Metrics Using Hybrid Topic Models

2020 ◽  
Vol 25 (6) ◽  
pp. 755-769
Author(s):  
Noorullah R. Mohammed ◽  
Moulana Mohammed
Keyword(s):  
Data Clustering ◽  
Topic Models ◽  
Cluster Validity ◽  
Text Documents ◽  
Text Data ◽  
Validity Assessment ◽  
Text Document ◽  
Cluster Validity Indices ◽  
Validity Indices ◽  
Data Clusters

Text data clustering is performed for organizing the set of text documents into the desired number of coherent and meaningful sub-clusters. Modeling the text documents in terms of topics derivations is a vital task in text data clustering. Each tweet is considered as a text document, and various topic models perform modeling of tweets. In existing topic models, the clustering tendency of tweets is assessed initially based on Euclidean dissimilarity features. Cosine metric is more suitable for more informative assessment, especially of text clustering. Thus, this paper develops a novel cosine based external and interval validity assessment of cluster tendency for improving the computational efficiency of tweets data clustering. In the experimental, tweets data clustering results are evaluated using cluster validity indices measures. Experimentally proved that cosine based internal and external validity metrics outperforms the other using benchmarked and Twitter-based datasets.

Incorporating Text OLAP in Business Intelligence

2011 ◽  
pp. 77-101 ◽  
Author(s):  
Byung-Kwon Park ◽  
Il-Yeol Song
Keyword(s):  
Information Retrieval ◽  
Text Mining ◽  
Business Intelligence ◽  
Multidimensional Analysis ◽  
Web Pages ◽  
Data Types ◽  
Text Documents ◽  
Text Data ◽  
Platform Architecture ◽  
Unstructured Text

As the amount of data grows very fast inside and outside of an enterprise, it is getting important to seamlessly analyze both data types for total business intelligence. The data can be classified into two categories: structured and unstructured. For getting total business intelligence, it is important to seamlessly analyze both of them. Especially, as most of business data are unstructured text documents, including the Web pages in Internet, we need a Text OLAP solution to perform multidimensional analysis of text documents in the same way as structured relational data. We first survey the representative works selected for demonstrating how the technologies of text mining and information retrieval can be applied for multidimensional analysis of text documents, because they are major technologies handling text data. And then, we survey the representative works selected for demonstrating how we can associate and consolidate both unstructured text documents and structured relation data for obtaining total business intelligence. Finally, we present a future business intelligence platform architecture as well as related research topics. We expect the proposed total heterogeneous business intelligence architecture, which integrates information retrieval, text mining, and information extraction technologies all together, including relational OLAP technologies, would make a better platform toward total business intelligence.

scholarly journals Improving the Decision Value of Hierarchical Text Clustering Using Term Overlap Detection

2015 ◽  
Vol 19 ◽  
Author(s):  
Nilupulee Nathawitharana ◽  
Damminda Alahakoon ◽  
Sumith Matharage
Keyword(s):  
Hierarchical Clustering ◽  
Categorical Data ◽  
Text Clustering ◽  
Written Language ◽  
Text Documents ◽  
Text Data ◽  
Text Document ◽  
Cluster Accuracy ◽  
Document Collection ◽  
A New Technique

Humans are used to expressing themselves with written language and language provides a medium with which we can describe our experiences in detail incorporating individuality. Even though documents provide a rich source of information, it becomes very difficult to identify, extract, summarize and search when vast amounts of documents are collected especially over time. Document clustering is a technique that has been widely used to group documents based on similarity of content represented by the words used. Once key groups are identified further drill down into sub-groupings is facilitated by the use of hierarchical clustering. Clustering and hierarchical clustering are very useful when applied to numerical and categorical data and cluster accuracy and purity measures exist to evaluate the outcomes of a clustering exercise. Although the same measures have been applied to text clustering, text clusters are based on words or terms which can be repeated across documents associated with different topics. Therefore text data cannot be considered as a direct ‘coding’ of a particular experience or situation in contrast to numerical and categorical data and term overlap is a very common characteristic in text clustering. In this paper we propose a new technique and methodology for term overlap capture from text documents, highlighting the different situations such overlap could signify and discuss why such understanding is important for obtaining value from text clustering. Experiments were conducted using a widely used text document collection where the proposed methodology allowed exploring the term diversity for a given document collection and obtain clusters with minimum term overlap.

scholarly journals Music Emotion Classification based on Lyrics-Audio using Corpus based Emotion

2018 ◽  
Vol 8 (3) ◽  
pp. 1720 ◽  
Author(s):  
Fika Hastarita Rachman ◽  
Riyanarto Sarno ◽  
Chastine Fatichah
Keyword(s):  
Audio Signal ◽  
Extraction Process ◽  
Emotion Classification ◽  
Text Documents ◽  
Text Data ◽  
Text Document ◽  
Audio Features ◽  
Test Result ◽  
F Measure

Music has lyrics and audio. That’s components can be a feature for music emotion classification. Lyric features were extracted from text data and audio features were extracted from audio signal data.In the classification of emotions, emotion corpus is required for lyrical feature extraction. Corpus Based Emotion (CBE) succeed to increase the value of F-Measure for emotion classification on text documents. The music document has an unstructured format compared with the article text document. So it requires good preprocessing and conversion process before classification process. We used MIREX Dataset for this research. Psycholinguistic and stylistic features were used as lyrics features. Psycholinguistic feature was a feature that related to the category of emotion. In this research, CBE used to support the extraction process of psycholinguistic feature. Stylistic features related with usage of unique words in the lyrics, e.g. ‘ooh’, ‘ah’, ‘yeah’, etc. Energy, temporal and spectrum features were extracted for audio features.The best test result for music emotion classification was the application of Random Forest methods for lyrics and audio features. The value of F-measure was 56.8%.

Text Mining-Machine Learning on Documents

2011 ◽  
pp. 1109-1112 ◽  
Author(s):  
Dunja Mladenic
Keyword(s):  
Text Mining ◽  
User Profiling ◽  
Mining Machine ◽  
Text Data ◽  
Automatic Data ◽  
Topic Identification ◽  
Document Search ◽  
Text Information ◽  
Document Categorization ◽  
Automatic Document Summarization

Intensive usage and growth of the World Wide Web and the daily increasing amount of text information in electronic form have resulted in a growing need for computer-supported ways of dealing with text data. One of the most popular problems addressed with text mining methods is document categorization. Document categorization aims to classify documents into pre-defined categories, based on their content. Other important problems addressed in text mining include document search, based on the content, automatic document summarization, automatic document clustering and construction of document hierarchies, document authorship detection, identification of plagiarism of documents, topic identification and tracking, information extraction, hypertext analysis, and user profiling. If we agree on text mining being a fairly broad area dealing with computer-supported analysis of text, then the list of problems that can be addressed is rather long and open. Here we adopt this fairly open view but concentrate on the parts related to automatic data analysis and data mining.

scholarly journals Aplikasi Text Mining untuk Klasterisasi Aduan Masyarakat Kota Semarang Menggunakan Algoritma K-means

2021 ◽  
Vol 18 (2) ◽  
pp. 215
Author(s):  
Dita Afida ◽  
Erika Devi Udayanti ◽  
Etika Kartikadarma
Keyword(s):  
Social Media ◽  
Text Mining ◽  
Clustering Algorithm ◽  
City Government ◽  
Community Based ◽  
Text Data ◽  
Government Organizations ◽  
Text Document ◽  
Data Source ◽  
Mining Methods

<p>Social media is a service that is very supportive for government activities, especially in providing openness and community-based government. One form of its implementation is the Semarang City government through the Center for Community Complaints Management (P3M), whose task is to manage community complaints that enter one of the communication channels namely social media twitter. The number of public complaints that enter every day is very varied. This is certainly quite difficult for managers in categorizing complaints reports according to the relevant Local Government Organizations (OPD). This paper focuses on the problem of how to conduct clustering of community complaints. The data source comes from Twitter using the keyword "Laporhendi". Text document data from community complaint tweets was analyzed by text mining methods. A number of pre-processing of text data processing begins with the process of case folding, tokenizing, stemming, stopword removal and word robbering with tf-idf. In conducting cluster mapping, clustering algorithm will be used in dividing the complaint cluster, namely the k-means algorithm. Evaluation of cluster results is done by using purity to determine the accuracy of the results of grouping or clustering.</p>

scholarly journals Exploring technology opportunities and evolution of IoT-related logistics services with text mining

2021 ◽  
Author(s):  
Mu-Chen Chen ◽  
Pui Hung Ho
Keyword(s):  
Text Mining ◽  
Text Documents ◽  
Text Data ◽  
Substantial Impact ◽  
Logistics Industry ◽  
Evolution Analysis ◽  
Technology Service ◽  
Service Evolution ◽  
Logistics Services ◽  
Methodological Guidelines

AbstractMany IoT technologies have been applied in the logistics industry in recent years, and they have had a substantial impact on many sectors such as shipping, air freight, warehousing, inventory, etc. Exploring technology opportunities and carrying out technological trend analysis are essential for IoT’s evolution, and there are many techniques or methods for doing so. In this paper, data analysis and text mining techniques, technology opportunity analysis (TOA) and technology-service evolution analysis (TSEA) have been applied to analyze and observe IoT technologies’ and services’ evolution. Academic journals, market reports, and patents have been collected and reviewed on the topic of IoT in the logistics field in this paper. Moreover, by using TOA, technology opportunities have been analyzed to explore IoT-related logistics services. The results of TOA, for example, show that cloud technology is essential to develop smart logistics services, and communication RFID technologies are key to developing information logistics services. Finally, TSEA enables the observation of IoT technology and logistics service evolution by combining unstructured and semi-structured data from text documents. Observing the results of TSEA, the evolution of IoT in logistics is identified, and the results of TSEA also confirm those of TOA using unstructured or semi-structured text data from documents only. The results of this paper are discussed and compared with those of some previous review studies. In summary, the results of this paper provide methodological guidelines on this topic for a comprehensive understanding of IoT-related logistics services.

scholarly journals LSA & LDA topic modeling classification: comparison study on e-books

2020 ◽  
Vol 19 (1) ◽  
pp. 353
Author(s):  
Shaymaa H. Mohammed ◽  
Salam Al-augby
Keyword(s):  
Digital Libraries ◽  
Full Text ◽  
Topic Modeling ◽  
Comparison Study ◽  
Text Documents ◽  
Text Data ◽  
Text Document ◽  
Unstructured Text ◽  
The One ◽  
Text Document Classification

<p>With the rapid growth of information technology, the amount of unstructured text data in digital libraries is rapidly increased and has become a big challenge in analyzing, organizing and how to classify text automatically in E-research repository to get the benefit from them is the cornerstone. The manual categorization of text documents requires a lot of financial, human resources for management. In order to get so, topic modeling are used to classify documents. This paper addresses a comparison study on scientific unstructured text document classification (e-books) based on the full text where applying the most popular topic modeling approach (LDA, LSA) to cluster the words into a set of topics as important keywords for classification. Our dataset consists of (300) books contain about 23 million words based on full text. In the used topic models (LSA, LDA) each word in the corpus of vocabulary is connected with one or more topics with a probability, as estimated by the model. Many (LDA, LSA) models were built with different values of coherence and pick the one that produces the highest coherence value. The result of this paper showed that LDA has better results than LSA and the best results obtained from the LDA method was (<strong>0.592179</strong>) of coherence value when the number of topics was <strong>20 while</strong> the LSA coherence value was <strong>(0.5773026)</strong> when the number of topics was 10.</p>

scholarly journals THE INFLUENCE OF TEXT PREPROCESSING METHODS AND TOOLS ON CALCULATING TEXT SIMILARITY

2019 ◽  
pp. 973
Author(s):  
Đorđe Petrović ◽  
Milena Stanković
Keyword(s):  
Text Mining ◽  
Vector Space ◽  
Vector Space Model ◽  
Text Similarity ◽  
Text Documents ◽  
Space Model ◽  
Text Document ◽  
The Subject ◽  
Text Preprocessing ◽  
Multidimensional Representation

Text mining to a great extent depends on the various text preprocessing techniques. The preprocessing methods and tools which are used to prepare texts for further mining can be divided into those which are and those which are not language-dependent. The subject matter of this research was the analysis of the influence of these methods and tools on further text mining. We first focused on the analysis of the influence on the reduction of the vector space model for the multidimensional represen-tation of text documents. We then analyzed the influence on calculating text similarity, which is the focus of this research. The conclusion we reached is that the implemen-tation of various text preprocessing methods in the Serbian language, which are used for the reduction of the vector space model for the multidimensional representation of text document, achieves the required results. But, the implementation of various text preprocessing methods specific to the Serbian language for the purpose of calculating text similarity can lead to great differences in the results.

Sign in / Sign up

Export Citation Format

Share Document