Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It

2017 ◽  
Author(s):  
Matthew James Denny ◽  
Arthur Spirling
Keyword(s):  
Unsupervised Learning ◽  
Text Preprocessing

Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It

2018 ◽  
Vol 26 (2) ◽  
pp. 168-189 ◽  
Author(s):  
Matthew J. Denny ◽  
Arthur Spirling
Keyword(s):  
Feature Selection ◽  
Unsupervised Learning ◽  
Political Science ◽  
Real Data ◽  
Statistical Procedure ◽  
Science Text ◽  
Substantive Theory ◽  
Text Preprocessing

Despite the popularity of unsupervised techniques for political science text-as-data research, the importance and implications of preprocessing decisions in this domain have received scant systematic attention. Yet, as we show, such decisions have profound effects on the results of real models for real data. We argue that substantive theory is typically too vague to be of use for feature selection, and that the supervised literature is not necessarily a helpful source of advice. To aid researchers working in unsupervised settings, we introduce a statistical procedure and software that examines the sensitivity of findings under alternate preprocessing regimes. This approach complements a researcher’s substantive understanding of a problem by providing a characterization of the variability changes in preprocessing choices may induce when analyzing a particular dataset. In making scholars aware of the degree to which their results are likely to be sensitive to their preprocessing decisions, it aids replication efforts.

scholarly journals Talk Show Segmentation System Based on Twitter Using K-Medoids Clustering Algorithm

2020 ◽  
Vol 3 (3) ◽  
pp. 158-163
Author(s):  
Kharisma Jevi Shafira Sepyanto ◽  
Yulison Herry Chrisnanto ◽  
Fajri Rakhmat Umbara
Keyword(s):  
Public Opinion ◽  
Unsupervised Learning ◽  
Clustering Algorithm ◽  
Talk Show ◽  
Analysis Process ◽  
Silhouette Coefficient ◽  
Competent Person ◽  
Long Time ◽  
Text Preprocessing ◽  
Coefficient Method

Innovations on a talk show on television can be a threat. Audience will be divided into groups so that it can make a downgrade rating program. Program ratings affect companies that will use advertising services. Television companies will go bankrupt. The biggest source of income is sales of advertising services. One way to overcome them can be analyzed in public opinion. The results of the analysis can provide information about the attractiveness of the community towards the program. But the analysis process takes a long time and can be done only by a competent person so another process is needed to get the results of the analysis that is fast and can be done by anyone. In this study using K-Medoids Clustering in the process of identifying public opinion. The clustering process known as unsupervised learning will be combined with the labeling process. The previous episode's tweet data will be labeled and then used to obtain the predicted labels from other cluster members. Before going through the clustering stage, the tweet data will go through the text preprocessing stage then transformed into a numeric form based on the appearance of the word. Transformation data will be clustered by calculating proximity using Cosine Similarity. Labels from the Medoids cluster will be used on unlabeled tweet data. The cluster results were tested using the Silhouette Coefficient method to get 0.19 results. However, this method successfully predicted public opinion and achieved an accuracy of 80%.

Classification of Observations through Combination of the Dimension Reduction and the Cluster Analysis

2017 ◽  
Vol 7 (8) ◽  
pp. 30
Author(s):  
Hyeuk Kim
Keyword(s):  
Machine Learning ◽  
Principal Component Analysis ◽  
Cluster Analysis ◽  
Unsupervised Learning ◽  
Principal Component ◽  
Component Analysis ◽  
Baseball Players ◽  
Partitioning Around Medoids ◽  
Different Characteristics

Unsupervised learning in machine learning divides data into several groups. The observations in the same group have similar characteristics and the observations in the different groups have the different characteristics. In the paper, we classify data by partitioning around medoids which have some advantages over the k-means clustering. We apply it to baseball players in Korea Baseball League. We also apply the principal component analysis to data and draw the graph using two components for axis. We interpret the meaning of the clustering graphically through the procedure. The combination of the partitioning around medoids and the principal component analysis can be used to any other data and the approach makes us to figure out the characteristics easily.

Klasifikasi pada Tempat Tinggal Menurut Provinsi dan Jenis Kepemilikan Berdasarkan Algoritma K-Means

2020 ◽  
Vol 4 (3) ◽  
pp. 247
Author(s):  
Dwi Swasono Rachmad
Keyword(s):  
Data Mining ◽  
Unsupervised Learning ◽  
Residential Buildings ◽  
Government Agency ◽  
Role Of Government ◽  
The Republic ◽  
Household Processing ◽  
Central Statistics

<p><em>H</em><em>ousing is derived from the word house</em><em> which means</em><em> a place that has a place to live which will stay or stop in a certain time. Housing is a residence that has been grouped into a place that has facilities and infrastructure. The problem in this study focuses on the type of residential ownership in the form of SHM ART, SHM Non ART, NON SHM and others. </em><em>T</em><em>hese four types</em><em> can be used</em><em> to know the percentage of ownership in all provinces in Indonesia. Due to the fact that there is still a lot of information about the type of certificate ownership, there is still not much ownership. Therefore, the use of the k-Means algorithm as a data mining concept in the form of clusters, where the data already has parameters or values that fall into the category of unsupervised learning. That data produced the best. The data was obtained from published sources of the Republic of Indonesia government agency, namely the Central Statistics Agency data with the category of household processing with self-owned residential buildings purchased from developers or non-developers by province and type of ownership in 2016 throughout Indonesia. In conducting the dataset, researchers used the RapidMiner application as a clustering process application. This research </em><em>shows that</em><em> there are more types of ownership in the SHM ART, but for other values it is still smaller than the value in other types of ownership which is the second largest value. So</em><em>,</em><em> in this case, the role of government in providing assistance in the process of ownership in order to become SHM ART</em><em> is very important</em><em>.</em></p>

Application of Machine Learning in Animal Disease Analysis and Prediction

2020 ◽  
Vol 15 ◽  
Author(s):  
Shuwen Zhang ◽  
Qiang Su ◽  
Qin Chen
Keyword(s):  
Machine Learning ◽  
Unsupervised Learning ◽  
Supervised Learning ◽  
Clustering Algorithm ◽  
Principal Component ◽  
Support Vector ◽  
Animal Disease ◽  
Human Beings ◽  
Animal Diseases ◽  
Disease Analysis

Abstract: Major animal diseases pose a great threat to animal husbandry and human beings. With the deepening of globalization and the abundance of data resources, the prediction and analysis of animal diseases by using big data are becoming more and more important. The focus of machine learning is to make computers learn how to learn from data and use the learned experience to analyze and predict. Firstly, this paper introduces the animal epidemic situation and machine learning. Then it briefly introduces the application of machine learning in animal disease analysis and prediction. Machine learning is mainly divided into supervised learning and unsupervised learning. Supervised learning includes support vector machines, naive bayes, decision trees, random forests, logistic regression, artificial neural networks, deep learning, and AdaBoost. Unsupervised learning has maximum expectation algorithm, principal component analysis hierarchical clustering algorithm and maxent. Through the discussion of this paper, people have a clearer concept of machine learning and understand its application prospect in animal diseases.

Sign in / Sign up

Export Citation Format

Share Document