scholarly journals WEClustering: word embeddings based text clustering technique for large datasets

2021 ◽  
Author(s):  
Vivek Mehta ◽  
Seema Bawa ◽  
Jasmeet Singh
Keyword(s):  
Text Clustering ◽  
High Dimensionality ◽  
Experimental Comparison ◽  
Adjusted Rand Index ◽  
Word Embeddings ◽  
Agglomerative Clustering ◽  
Data Mining Technique ◽  
Clustering Techniques ◽  
Clustering Technique ◽  
Effective Manner

AbstractA massive amount of textual data now exists in digital repositories in the form of research articles, news articles, reviews, Wikipedia articles, and books, etc. Text clustering is a fundamental data mining technique to perform categorization, topic extraction, and information retrieval. Textual datasets, especially which contain a large number of documents are sparse and have high dimensionality. Hence, traditional clustering techniques such as K-means, Agglomerative clustering, and DBSCAN cannot perform well. In this paper, a clustering technique especially suitable to large text datasets is proposed that overcome these limitations. The proposed technique is based on word embeddings derived from a recent deep learning model named “Bidirectional Encoders Representations using Transformers”. The proposed technique is named as WEClustering. The proposed technique deals with the problem of high dimensionality in an effective manner, hence, more accurate clusters are formed. The technique is validated on several datasets of varying sizes and its performance is compared with other widely used and state of the art clustering techniques. The experimental comparison shows that the proposed clustering technique gives a significant improvement over other techniques as measured by metrics such Purity and Adjusted Rand Index.

scholarly journals Clustering Techniques for Secondary Substations Siting

Energies ◽  
2021 ◽  
Vol 14 (4) ◽  
pp. 1028
Author(s):  
Silvia Corigliano ◽  
Federico Rosato ◽  
Carla Ortiz Dominguez ◽  
Marco Merlo
Keyword(s):  
Rural Areas ◽  
Urban Areas ◽  
Universal Access ◽  
Distribution Networks ◽  
Industrialized Countries ◽  
Agglomerative Clustering ◽  
Clustering Techniques ◽  
Hierarchical Agglomerative Clustering ◽  
Efficient Planning ◽  
Target Set

The scientific community is active in developing new models and methods to help reach the ambitious target set by UN SDGs7: universal access to electricity by 2030. Efficient planning of distribution networks is a complex and multivariate task, which is usually split into multiple subproblems to reduce the number of variables. The present work addresses the problem of optimal secondary substation siting, by means of different clustering techniques. In contrast with the majority of approaches found in the literature, which are devoted to the planning of MV grids in already electrified urban areas, this work focuses on greenfield planning in rural areas. K-means algorithm, hierarchical agglomerative clustering, and a method based on optimal weighted tree partitioning are adapted to the problem and run on two real case studies, with different population densities. The algorithms are compared in terms of different indicators useful to assess the feasibility of the solutions found. The algorithms have proven to be effective in addressing some of the crucial aspects of substations siting and to constitute relevant improvements to the classic K-means approach found in the literature. However, it is found that it is very challenging to conjugate an acceptable geographical span of the area served by a single substation with a substation power high enough to justify the installation when the load density is very low. In other words, well known standards adopted in industrialized countries do not fit with developing countries’ requirements.

scholarly journals Clustering electricity market participants via FRM models

2020 ◽  
pp. 1-12
Author(s):  
Ayla Gülcü ◽  
Sedrettin Çalişkan
Keyword(s):  
Electricity Market ◽  
Distance Metrics ◽  
Agglomerative Clustering ◽  
Timely Manner ◽  
Short Term ◽  
Clustering Techniques ◽  
Market Participants ◽  
Cosine Distance ◽  
Collateral Mechanism

Collateral mechanism in the Electricity Market ensures the payments are executed on a timely manner; thus maintains the continuous cash flow. In order to value collaterals, Takasbank, the authorized central settlement bank, creates segments of the market participants by considering their short-term and long-term debt/credit information arising from all market activities. In this study, the data regarding participants’ daily and monthly debt payment and penalty behaviors is analyzed with the aim of discovering high-risk participants that fail to clear their debts on-time frequently. Different clustering techniques along with different distance metrics are considered to obtain the best clustering. Moreover, data preprocessing techniques along with Recency, Frequency, Monetary Value (RFM) scoring have been used to determine the best representation of the data. The results show that Agglomerative Clustering with cosine distance achieves the best separated clustering when the non-normalized dataset is used; this is also acknowledged by a domain expert.

scholarly journals Embed2Detect: temporally clustered embedded words for event detection in social media

2021 ◽  
Author(s):  
Hansi Hettiarachchi ◽  
Mariam Adedoyin-Olowe ◽  
Jagdev Bhogal ◽  
Mohamed Medhat Gaber
Keyword(s):  
Social Media ◽  
Event Detection ◽  
High Volume ◽  
Detection Methods ◽  
Word Embeddings ◽  
Agglomerative Clustering ◽  
Data Set ◽  
Social Media Data ◽  
Social Media Platforms ◽  
Media Data

AbstractSocial media is becoming a primary medium to discuss what is happening around the world. Therefore, the data generated by social media platforms contain rich information which describes the ongoing events. Further, the timeliness associated with these data is capable of facilitating immediate insights. However, considering the dynamic nature and high volume of data production in social media data streams, it is impractical to filter the events manually and therefore, automated event detection mechanisms are invaluable to the community. Apart from a few notable exceptions, most previous research on automated event detection have focused only on statistical and syntactical features in data and lacked the involvement of underlying semantics which are important for effective information retrieval from text since they represent the connections between words and their meanings. In this paper, we propose a novel method termed Embed2Detect for event detection in social media by combining the characteristics in word embeddings and hierarchical agglomerative clustering. The adoption of word embeddings gives Embed2Detect the capability to incorporate powerful semantical features into event detection and overcome a major limitation inherent in previous approaches. We experimented our method on two recent real social media data sets which represent the sports and political domain and also compared the results to several state-of-the-art methods. The obtained results show that Embed2Detect is capable of effective and efficient event detection and it outperforms the recent event detection methods. For the sports data set, Embed2Detect achieved 27% higher F-measure than the best-performed baseline and for the political data set, it was an increase of 29%.

scholarly journals Hybrid SVD For Document Representation Using Different Vectorization

2019 ◽  
pp. 388-393
Author(s):  
Kalpana P ◽  
Rosini B R ◽  
Sathya Priya K P ◽  
Sowmiya S
Keyword(s):  
Document Clustering ◽  
High Dimensionality ◽  
Word Embeddings ◽  
Document Representation ◽  
Challenging Problem ◽  
Large Database ◽  
Relevant Document ◽  
Electronic Form ◽  
Large Databases ◽  
Uniform Manner

Document Clustering is the process of segmenting a particular collection of text into subgroups. Nowadays all documents are in electronic form, because of the issue to retrieve relevant document from the large database. The goal is to transform text composed of daily language in a structured, database format. In this way, different documents are summarized and presented in a uniform manner. The challenging problem of document clustering are big volume, high dimensionality and complex semantics. The objective of this paper is mainly focused on clustering multi-sense word embeddings using three different algorithms(K-means, DBSCAN, CURE). Among these three algorithm CURE gives better accuracy and it can handle large databases efficiently.</p>

scholarly journals Identifying Network Anomalies Using Clustering Technique in Weblog Data

2012 ◽  
Vol 2 (3) ◽  
pp. 71-73 ◽  
Author(s):  
Kiran Bejjanki ◽  
A. Bhaskar
Keyword(s):  
Network Flow ◽  
Flow Data ◽  
Ip Address ◽  
Clustering Techniques ◽  
Clustering Technique ◽  
Network Anomalies

In this paper we present an approach for identifying networkanomalies by visualizing network flow data which is stored inweblogs. Various clustering techniques can be used to identifydifferent anomalies in the network. Here, we present a newapproach based on simple K-Means for analyzing networkflow data using different attributes like IP address, Protocol,Port number etc. to detect anomalies. By using visualization,we can identify which sites are more frequently accessed bythe users. In our approach we provide overview about givendataset by studying network key parameters. In this processwe used preprocessing techniques to eliminate unwantedattributes from weblog data.

scholarly journals Detection of Crimes Using Unsupervised Learning Techniques

2020 ◽  
Vol 2 (1) ◽  
pp. 8-11
Author(s):  
R. Buli Babu ◽  
G. Snehal ◽  
Aditya Satya Kiran
Keyword(s):  
Data Mining ◽  
Unsupervised Learning ◽  
Expectation Maximization ◽  
Expectation Maximization Algorithm ◽  
Similar Data ◽  
Data Mining Technique ◽  
Clustering Techniques ◽  
Mining Technique ◽  
Learning Techniques ◽  
Object Based

Data mining can be used to detect model crime problems. This paper is about the importance of datamining about its techniques and how we can easily solve the crime. Crime data will be stored in criminal’s database.To analyze the data easily we have data mining technique that is clustering. Clustering is a method to group identicalcharacteristics in which the similarity is maximized or minimized. In clustering techniques also we have different typeof algorithm, but in this paper we are using the k-means algorithm and expectation-maximization algorithm. We areusing these techniques because these two techniques come under the partition algorithm. Partition algorithm is oneof the best methods to solve crimes and to find the similar data and group it. K-means algorithm is used to partitionthe grouped object based on their means. Expectation-maximization algorithm is the extension of k-means algorithmhere we partition the data based on their parameters.

Sign in / Sign up

Export Citation Format

Share Document