scholarly journals The Efficient Storage of Text Documents in Digital Libraries

2009 ◽  
Vol 28 (3) ◽  
pp. 143 ◽  
Author(s):  
Przemyslaw Skibiński ◽  
Jakub Swacha
Keyword(s):  
Data Compression ◽  
Digital Libraries ◽  
Text Compression ◽  
Storage Space ◽  
Software Library ◽  
Source Text ◽  
Text Documents ◽  
Space Reduction ◽  
Text Document ◽  
Efficient Storage

In this paper we investigate the possibility of improving the efficiency of data compression, and thus reducing storage requirements, for seven widely used text document formats. We propose an open-source text compression software library, featuring an advanced word-substitution scheme with static and semidynamic word dictionaries. The empirical results show an average storage space reduction as high as 78 percent compared to uncompressed documents, and as high as 30 percent compared to documents compressed with the free compression software gzip.

scholarly journals Topic Modeling Based Extractive Text Summarization

2020 ◽  
Vol 9 (4) ◽  
pp. 1710-1719
Keyword(s):  
Topic Modeling ◽  
State Of The Art ◽  
Text Summarization ◽  
Computational Technique ◽  
Source Text ◽  
Text Documents ◽  
Text Document ◽  
Current State ◽  
Modeling Techniques ◽  
Novel Method

Text summarization is an approach for identifying important information present within text documents. This computational technique aims to generate shorter versions of the source text, by including only the relevant and salient information present within the source text. In this paper, we propose a novel method to summarize a text document by clustering its contents based on latent topics produced using topic modeling techniques and by generating extractive summaries for each of the identified text clusters. All extractive sub-summaries are later combined to generate a summary for any given source document. We utilize the lesser used and challenging WikiHow dataset in our approach to text summarization. This dataset is unlike the commonly used news datasets which are available for text summarization. The well-known news datasets present their most important information in the first few lines of their source texts, which make their summarization a lesser challenging task when compared to summarizing the WikiHow dataset. Contrary to these news datasets, the documents in the WikiHow dataset are written using a generalized approach and have lesser abstractedness and higher compression ratio, thus proposing a greater challenge to generate summaries. A lot of the current state-of-the-art text summarization techniques tend to eliminate important information present in source documents in the favor of brevity. Our proposed technique aims to capture all the varied information present in source documents. Although the dataset proved challenging, after performing extensive tests within our experimental setup, we have discovered that our model produces encouraging ROUGE results and summaries when compared to the other published extractive and abstractive text summarization models

scholarly journals LSA & LDA topic modeling classification: comparison study on e-books

2020 ◽  
Vol 19 (1) ◽  
pp. 353
Author(s):  
Shaymaa H. Mohammed ◽  
Salam Al-augby
Keyword(s):  
Digital Libraries ◽  
Full Text ◽  
Topic Modeling ◽  
Comparison Study ◽  
Text Documents ◽  
Text Data ◽  
Text Document ◽  
Unstructured Text ◽  
The One ◽  
Text Document Classification

<p>With the rapid growth of information technology, the amount of unstructured text data in digital libraries is rapidly increased and has become a big challenge in analyzing, organizing and how to classify text automatically in E-research repository to get the benefit from them is the cornerstone. The manual categorization of text documents requires a lot of financial, human resources for management. In order to get so, topic modeling are used to classify documents. This paper addresses a comparison study on scientific unstructured text document classification (e-books) based on the full text where applying the most popular topic modeling approach (LDA, LSA) to cluster the words into a set of topics as important keywords for classification. Our dataset consists of (300) books contain about 23 million words based on full text. In the used topic models (LSA, LDA) each word in the corpus of vocabulary is connected with one or more topics with a probability, as estimated by the model. Many (LDA, LSA) models were built with different values of coherence and pick the one that produces the highest coherence value. The result of this paper showed that LDA has better results than LSA and the best results obtained from the LDA method was (<strong>0.592179</strong>) of coherence value when the number of topics was <strong>20 while</strong> the LSA coherence value was <strong>(0.5773026)</strong> when the number of topics was 10.</p>

An Improved B-hill Climbing Optimization Technique for Solving the Text Documents Clustering Problem

2020 ◽  
Vol 16 (4) ◽  
pp. 296-306 ◽  
Author(s):  
Laith Mohammad Abualigah ◽  
Essam Said Hanandeh ◽  
Ahamad Tajudin Khader ◽  
Mohammed Abdallh Otair ◽  
Shishir Kumar Shandilya
Keyword(s):  
Optimization Technique ◽  
Document Clustering ◽  
Text Clustering ◽  
Hill Climbing ◽  
Text Documents ◽  
Clustering Problem ◽  
Text Document ◽  
Text Information ◽  
Amount Of Knowledge ◽  
The Hill

Background: Considering the increasing volume of text document information on Internet pages, dealing with such a tremendous amount of knowledge becomes totally complex due to its large size. Text clustering is a common optimization problem used to manage a large amount of text information into a subset of comparable and coherent clusters. Aims: This paper presents a novel local clustering technique, namely, β-hill climbing, to solve the problem of the text document clustering through modeling the β-hill climbing technique for partitioning the similar documents into the same cluster. Methods: The β parameter is the primary innovation in β-hill climbing technique. It has been introduced in order to perform a balance between local and global search. Local search methods are successfully applied to solve the problem of the text document clustering such as; k-medoid and kmean techniques. Results: Experiments were conducted on eight benchmark standard text datasets with different characteristics taken from the Laboratory of Computational Intelligence (LABIC). The results proved that the proposed β-hill climbing achieved better results in comparison with the original hill climbing technique in solving the text clustering problem. Conclusion: The performance of the text clustering is useful by adding the β operator to the hill climbing.

Design and Implementation of Low Energy Wireless Network Nodes based on Hardware Compression Acceleration

2019 ◽  
Vol 12 ◽  
Author(s):  
Hui Yang ◽  
Anand Nayyar
Keyword(s):  
Energy Consumption ◽  
Data Compression ◽  
Energy Saving ◽  
Optimization Design ◽  
Hardware Acceleration ◽  
Transmission Efficiency ◽  
General Purpose ◽  
Storage Space ◽  
General Purpose Processor ◽  
Compression Time

: In the fast development of information, the information data is increasing in geometric multiples, and the speed of information transmission and storage space are required to be higher. In order to reduce the use of storage space and further improve the transmission efficiency of data, data need to be compressed. processing. In the process of data compression, it is very important to ensure the lossless nature of data, and lossless data compression algorithms appear. The gradual optimization design of the algorithm can often achieve the energy-saving optimization of data compression. Similarly, The effect of energy saving can also be obtained by improving the hardware structure of node. In this paper, a new structure is designed for sensor node, which adopts hardware acceleration, and the data compression module is separated from the node microprocessor.On the basis of the ASIC design of the algorithm, by introducing hardware acceleration, the energy consumption of the compressed data was successfully reduced, and the proportion of energy consumption and compression time saved by the general-purpose processor was as high as 98.4 % and 95.8 %, respectively. It greatly reduces the compression time and energy consumption.

The Design of Gas Real Time Monitoring System Based on Gas Alarm Sensor

2014 ◽  
Vol 556-562 ◽  
pp. 2940-2943
Author(s):  
Wei Dai ◽  
Gang Xie ◽  
Bai Qin Zhao
Keyword(s):  
Data Compression ◽  
Real Time ◽  
Monitoring System ◽  
Transmission Mechanism ◽  
Storage Space ◽  
Real Time Monitoring ◽  
Remote Computer ◽  
Data Store ◽  
Range Transport ◽  
Gas Environment

The gas alarms based on sensors are widely used, but there are still some limitations. By design a gas real time monitoring system, the gas alarms within a certain range transport collected data to a remote computer via wired or wireless method. Remote computer can monitoring the gas environment by receiving and monitoring alarms’ collected data. In order to ensure system’s reliability, transmission mechanism based on handshake is used. Data compression technology also included to reduce the storage space required by data store.

scholarly journals Development of the documents comparison module for an electronic document management system

2019 ◽  
pp. 527-533
Author(s):  
M A Mikheev ◽  
P Y Yakimov
Keyword(s):  
Character Recognition ◽  
Optical Character Recognition ◽  
Document Management ◽  
Electronic Document ◽  
Text Documents ◽  
Text Document ◽  
Document Management System ◽  
Optical Character ◽  
Electronic Document Management ◽  
Scanned Image

The article is devoted to solving the problem of document versions comparison in electronic document management systems. Systems-analogues were considered, the process of comparing text documents was studied. In order to recognize the text on the scanned image, the technology of optical character recognition and its implementation — Tesseract library were chosen. The Myers algorithm is applied to compare received texts. The software implementation of the text document comparison module was implemented using the solutions described above.

scholarly journals The Evaluation of Accuracy Performance in an Enhanced Embedded Feature Selection for Unstructured Text Classification

2020 ◽  
pp. 3397-3407
Author(s):  
Nur Syafiqah Mohd Nafis ◽  
Suryanti Awang
Keyword(s):  
Feature Selection ◽  
Text Classification ◽  
Training Dataset ◽  
Recursive Feature Elimination ◽  
High Dimensional ◽  
Significant Feature ◽  
Support Vector ◽  
Svm Classifier ◽  
Text Documents ◽  
Text Document

Text documents are unstructured and high dimensional. Effective feature selection is required to select the most important and significant feature from the sparse feature space. Thus, this paper proposed an embedded feature selection technique based on Term Frequency-Inverse Document Frequency (TF-IDF) and Support Vector Machine-Recursive Feature Elimination (SVM-RFE) for unstructured and high dimensional text classificationhis technique has the ability to measure the feature’s importance in a high-dimensional text document. In addition, it aims to increase the efficiency of the feature selection. Hence, obtaining a promising text classification accuracy. TF-IDF act as a filter approach which measures features importance of the text documents at the first stage. SVM-RFE utilized a backward feature elimination scheme to recursively remove insignificant features from the filtered feature subsets at the second stage. This research executes sets of experiments using a text document retrieved from a benchmark repository comprising a collection of Twitter posts. Pre-processing processes are applied to extract relevant features. After that, the pre-processed features are divided into training and testing datasets. Next, feature selection is implemented on the training dataset by calculating the TF-IDF score for each feature. SVM-RFE is applied for feature ranking as the next feature selection step. Only top-rank features will be selected for text classification using the SVM classifier. Based on the experiments, it shows that the proposed technique able to achieve 98% accuracy that outperformed other existing techniques. In conclusion, the proposed technique able to select the significant features in the unstructured and high dimensional text document.

scholarly journals Lossless text compression using GPT-2 language model and Huffman coding

2021 ◽  
Vol 102 ◽  
pp. 04013
Author(s):  
Md. Atiqur Rahman ◽  
Mohamed Hamada
Keyword(s):  
Data Compression ◽  
State Of The Art ◽  
Language Model ◽  
Huffman Coding ◽  
Original Text ◽  
Text Compression ◽  
Compression Technique ◽  
Daily Life Activities ◽  
Burrows Wheeler Transform ◽  
Compressed Data

Modern daily life activities produced lots of information for the advancement of telecommunication. It is a challenging issue to store them on a digital device or transmit it over the Internet, leading to the necessity for data compression. Thus, research on data compression to solve the issue has become a topic of great interest to researchers. Moreover, the size of compressed data is generally smaller than its original. As a result, data compression saves storage and increases transmission speed. In this article, we propose a text compression technique using GPT-2 language model and Huffman coding. In this proposed method, Burrows-Wheeler transform and a list of keys are used to reduce the original text file’s length. Finally, we apply GPT-2 language mode and then Huffman coding for encoding. This proposed method is compared with the state-of-the-art techniques used for text compression. Finally, we show that the proposed method demonstrates a gain in compression ratio compared to the other state-of-the-art methods.

Assessment of Twitter Data Clusters with Cosine-Based Validation Metrics Using Hybrid Topic Models

2020 ◽  
Vol 25 (6) ◽  
pp. 755-769
Author(s):  
Noorullah R. Mohammed ◽  
Moulana Mohammed
Keyword(s):  
Data Clustering ◽  
Topic Models ◽  
Cluster Validity ◽  
Text Documents ◽  
Text Data ◽  
Validity Assessment ◽  
Text Document ◽  
Cluster Validity Indices ◽  
Validity Indices ◽  
Data Clusters

Text data clustering is performed for organizing the set of text documents into the desired number of coherent and meaningful sub-clusters. Modeling the text documents in terms of topics derivations is a vital task in text data clustering. Each tweet is considered as a text document, and various topic models perform modeling of tweets. In existing topic models, the clustering tendency of tweets is assessed initially based on Euclidean dissimilarity features. Cosine metric is more suitable for more informative assessment, especially of text clustering. Thus, this paper develops a novel cosine based external and interval validity assessment of cluster tendency for improving the computational efficiency of tweets data clustering. In the experimental, tweets data clustering results are evaluated using cluster validity indices measures. Experimentally proved that cosine based internal and external validity metrics outperforms the other using benchmarked and Twitter-based datasets.

Sign in / Sign up

Export Citation Format

Share Document