scholarly journals Stream-Based Lossless Data Compression

2021 ◽  
pp. 391-410
Author(s):  
Shinichi Yamagiwa
Keyword(s):  
Data Compression ◽  
Data Streams ◽  
Data Stream ◽  
Optimization Techniques ◽  
Continuous Data ◽  
Massive Data ◽  
Compression Mechanism ◽  
Lossless Data Compression ◽  
Optimal Implementation

AbstractIn this chapter, we introduce aspects of applying data-compression techniques. First, we study the background of recent communication data paths. The focus of this chapter is a fast lossless data-compression mechanism that handles data streams completely. A data stream comprises continuous data with no termination of the massive data generated by sources such as movies and sensors. In this chapter, we introduce LCA-SLT and LCA-DLT, which accept the data streams, as well as several implementations of these stream-based compression techniques. We also show optimization techniques for optimal implementation in hardware.

Joint Source-Channel Coding in Dictionary Methods of Lossless Data Compression

2010 ◽  
Vol 56 (4) ◽  
pp. 351-355
Author(s):  
Marcin Rodziewicz
Keyword(s):  
Data Compression ◽  
Channel Coding ◽  
Data Stream ◽  
Noisy Channels ◽  
Error Resiliency ◽  
Joint Source Channel Coding ◽  
Lossless Data Compression ◽  
Compressed Data ◽  
Source Channel Coding

Joint Source-Channel Coding in Dictionary Methods of Lossless Data Compression Limitations on memory and resources of communications systems require powerful data compression methods. Decompression of compressed data stream is very sensitive to errors which arise during transmission over noisy channels, therefore error correction coding is also required. One of the solutions to this problem is the application of joint source and channel coding. This paper contains a description of methods of joint source-channel coding based on the popular data compression algorithms LZ'77 and LZSS. These methods are capable of introducing some error resiliency into compressed stream of data without degradation of the compression ratio. We analyze joint source and channel coding algorithms based on these compression methods and present their novel extensions. We also present some simulation results showing usefulness and achievable quality of the analyzed algorithms.

scholarly journals USE OF NEW EFFICIENT LOSSLESS DATA COMPRESSION METHOD IN TRANSMITTING ENCRYPTED BAPTISTA SYMMETRIC CHAOTIC CRYPTOSYSTEM DATA

2016 ◽  
Vol 78 (6-4) ◽  
Author(s):  
Muhamad Azlan Daud ◽  
Muhammad Rezal Kamel Ariffin ◽  
S. Kularajasingam ◽  
Che Haziqah Che Hussin ◽  
Nurliyana Juhan ◽  
...  
Keyword(s):  
Data Compression ◽  
Data Transmission ◽  
Compression Algorithm ◽  
Huffman Coding ◽  
Compression Method ◽  
Compression Technique ◽  
Chaotic Dynamical System ◽  
Fast Encoding ◽  
Compression Mechanism ◽  
Lossless Data Compression

A new compression algorithm used to ensure a modified Baptista symmetric cryptosystem which is based on a chaotic dynamical system to be applicable is proposed. The Baptista symmetric cryptosystem able to produce various ciphers responding to the same message input. This modified Baptista type cryptosystem suffers from message expansion that goes against the conventional methodology of a symmetric cryptosystem. A new lossless data compression algorithm based on theideas from the Huffman coding for data transmission is proposed.This new compression mechanism does not face the problem of mapping elements from a domain which is much larger than its range.Our new algorithm circumvent this problem via a pre-defined codeword list.  The purposed algorithm has fast encoding and decoding mechanism and proven analytically to be a lossless data compression technique.

Lossless Data Compression of Packet Data Streams,

1996 ◽  
Author(s):  
Junho Choi ◽  
Mitchell R. Grunes
Keyword(s):  
Data Compression ◽  
Data Streams ◽  
Packet Data ◽  
Lossless Data Compression

scholarly journals Resolving longitudinal amplitude and phase information of two continuous data streams for high-speed and real-time processing

2009 ◽  
Vol 7 ◽  
pp. 133-137 ◽  
Author(s):  
A. Guntoro ◽  
M. Glesner
Keyword(s):  
Data Streams ◽  
Data Stream ◽  
High Speed ◽  
Hardware Implementation ◽  
Software Implementation ◽  
Phase Detector ◽  
Continuous Data ◽  
Process Data ◽  
Real Time Processing ◽  
High Speed Data

Abstract. Although there is an increase of performance in DSPs, due to its nature of execution a DSP could not perform high-speed data processing on a continuous data stream. In this paper we discuss the hardware implementation of the amplitude and phase detector and the validation block on a FPGA. Contrary to the software implementation which can only process data stream as high as 1.5 MHz, the hardware approach is 225 times faster and introduces much less latency.

scholarly journals Stream-Based Lossless Data Compression Applying Adaptive Entropy Coding for Hardware-Based Implementation

Algorithms ◽  
2020 ◽  
Vol 13 (7) ◽  
pp. 159 ◽  
Author(s):  
Shinichi Yamagiwa ◽  
Eisaku Hayakawa ◽  
Koichi Marumo
Keyword(s):  
Big Data ◽  
Data Compression ◽  
Data Stream ◽  
High Speed ◽  
Data Communication ◽  
Streaming Data ◽  
Entropy Coding ◽  
Big Data Applications ◽  
Lossless Data Compression ◽  
Compressed Data

Toward strong demand for very high-speed I/O for processors, physical performance growth of hardware I/O speed was drastically increased in this decade. However, the recent Big Data applications still demand the larger I/O bandwidth and the lower latency for the speed. Because the current I/O performance does not improve so drastically, it is the time to consider another way to increase it. To overcome this challenge, we focus on lossless data compression technology to decrease the amount of data itself in the data communication path. The recent Big Data applications treat data stream that flows continuously and never allow stalling processing due to the high speed. Therefore, an elegant hardware-based data compression technology is demanded. This paper proposes a novel lossless data compression, called ASE coding. It encodes streaming data by applying the entropy coding approach. ASE coding instantly assigns the fewest bits to the corresponding compressed data according to the number of occupied entries in a look-up table. This paper describes the detailed mechanism of ASE coding. Furthermore, the paper demonstrates performance evaluations to promise that ASE coding adaptively shrinks streaming data and also works on a small amount of hardware resources without stalling or buffering any part of data stream.

scholarly journals Exception Handling Method Based on Event from Look-Up Table Applying Stream-Based Lossless Data Compression

Electronics ◽  
2021 ◽  
Vol 10 (3) ◽  
pp. 240
Author(s):  
Shinichi Yamagiwa ◽  
Koichi Marumo ◽  
Suzukaze Kuwabara
Keyword(s):  
Big Data ◽  
Data Compression ◽  
Original Data ◽  
Exception Handling ◽  
Continuous Data ◽  
Compression Process ◽  
Compression Performance ◽  
Look Up Table ◽  
Lossless Data Compression ◽  
Networking Technologies

It is getting popular to implement an environment where communications are performed remotely among IoT edge devices, such as sensory devices and the cloud servers due to applying, for example, artificial intelligence algorithms to the system. In such situations that handle big data, lossless data compression is one of the solutions to reduce the big data. In particular, the stream-based data compression technology is focused on such systems to compress infinitely continuous data stream with very small delay. However, during the continuous data compression process, it is not able to insert an exception code among the compressed data without any additional mechanisms, such as data framing and the packeting technique, as used in networking technologies. The exception code indicates configurations for the compressor/decompressor and/or its peripheral logics. Then, it is used in real time for the configuration of parameters against those components. To implement the exception code, data compression algorithm must include a mechanism to distinguish original data before compression and the exception code clearly. However, the conventional algorithms do not include such mechanism. This paper proposes novel methods to implement the exception code in data compression that uses look-up table, called the exception symbol. Additionally, we describe implementation details of the method by applying it to algorithms of stream-based data compression. Because some of the proposed mechanisms need to reserve entries in the table, we also discuss the effect against data compression performance according to experimental evaluations.

A State-of-the-Art Review of Data Stream Anonymization Schemes

2014 ◽  
pp. 24-50 ◽  
Author(s):  
Aderonke B. Sakpere ◽  
Anne V. D. M. Kayem
Keyword(s):  
Real Time ◽  
Data Streams ◽  
Data Stream ◽  
Data Privacy ◽  
Streaming Data ◽  
Continuous Data ◽  
Privacy Concerns ◽  
Useful Knowledge ◽  
Transient Nature ◽  
Address Data

Streaming data emerges from different electronic sources and needs to be processed in real time with minimal delay. Data streams can generate hidden and useful knowledge patterns when mined and analyzed. In spite of these benefits, the issue of privacy needs to be addressed before streaming data is released for mining and analysis purposes. In order to address data privacy concerns, several techniques have emerged. K-anonymity has received considerable attention over other privacy preserving techniques because of its simplicity and efficiency in protecting data. Yet, k-anonymity cannot be directly applied on continuous data (data streams) because of its transient nature. In this chapter, the authors discuss the challenges faced by k-anonymity algorithms in enforcing privacy on data streams and review existing privacy techniques for handling data streams.

Toward a Grid-Based Zero-Latency Data Warehousing Implementation for Continuous Data Streams Processing

2008 ◽  
pp. 755-786
Author(s):  
Tho Manh Nguyen ◽  
Peter Brezany ◽  
A. Min Tjoa ◽  
Edgar Weippl
Keyword(s):  
Knowledge Base ◽  
Data Streams ◽  
Data Stream ◽  
Resource Limitation ◽  
Large Data ◽  
High Volume ◽  
Continuous Data ◽  
Latency Data ◽  
Dynamic Collaboration ◽  
Grid Based

Continuous data streams are information sources in which data arrives in high volume in unpredictable rapid bursts. Processing data streams is a challenging task due to (1) the problem of random access to fast and large data streams using present storage technologies and (2) the exact answers from data streams often being too expensive. A framework of building a Grid-based Zero-Latency Data Stream Warehouse (GZLDSWH) to overcome the resource limitation issues in data stream processing without using approximation approaches is specified. The GZLDSWH is built upon a set of Open Grid Service Infrastructure (OGSI)-based services and Globus Toolkit 3 (GT3) with the capability of capturing and storing continuous data streams, performing analytical processing, and reacting autonomously in near real time to some kinds of events based on a well-established knowledge base. The requirements of a GZLDSWH, its Grid-based conceptual architecture, and the operations of its service are described in this paper. Furthermore, several challenges and issues in building a GZLDSWH, such as the Dynamic Collaboration Model between the Grid services, the Analytical Model, and the Design and Evaluation aspects of the Knowledge Base Rules are discussed and investigated.

scholarly journals Data Stream Classification Based on the Gamma Classifier

2015 ◽  
Vol 2015 ◽  
pp. 1-17 ◽  
Author(s):  
Abril Valeria Uriarte-Arcia ◽  
Itzamá López-Yáñez ◽  
Cornelio Yáñez-Márquez ◽  
João Gama ◽  
Oscar Camacho-Nieto
Keyword(s):  
Data Streams ◽  
Time Management ◽  
Data Stream ◽  
High Speed ◽  
Concept Drift ◽  
Synthetic Data ◽  
Continuous Data ◽  
Data Generation ◽  
Underlying Distribution ◽  
Data Stream Classification

The ever increasing data generation confronts us with the problem of handling online massive amounts of information. One of the biggest challenges is how to extract valuable information from these massive continuous data streams during single scanning. In a data stream context, data arrive continuously at high speed; therefore the algorithms developed to address this context must be efficient regarding memory and time management and capable of detecting changes over time in the underlying distribution that generated the data. This work describes a novel method for the task of pattern classification over a continuous data stream based on an associative model. The proposed method is based on the Gamma classifier, which is inspired by the Alpha-Beta associative memories, which are both supervised pattern recognition models. The proposed method is capable of handling the space and time constrain inherent to data stream scenarios. The Data Streaming Gamma classifier (DS-Gamma classifier) implements a sliding window approach to provide concept drift detection and a forgetting mechanism. In order to test the classifier, several experiments were performed using different data stream scenarios with real and synthetic data streams. The experimental results show that the method exhibits competitive performance when compared to other state-of-the-art algorithms.

Optimal Energy Management Strategy for Parallel Scheduling

2014 ◽  
Vol 568-570 ◽  
pp. 1539-1546
Author(s):  
Xin Li Li
Keyword(s):  
Data Streams ◽  
Power Efficiency ◽  
Data Stream ◽  
Large Scale ◽  
Massive Data ◽  
Energy Management Strategy ◽  
Parallel Scheduling ◽  
Large Scale Data ◽  
Scheduling Strategies ◽  
Scale Data

Large-scale data streams processing is now fundamental to many data processing applications. There is growing focus on manipulating Large-scale data streams on GPUs in order to improve the data throughput. Hence, there is a need to investigate the parallel scheduling strategy at the task level for the Large-scale data streamsprocessing, and to support them efficiently. We propose two different parallel scheduling strategies to handle massive data streamsin real time. Additionally, massive data streamsprocessing on GPUs is energy-consumed computation task. So we consider the power efficiency as an important factor to the parallel strategies. We present an approximation method to quantify the power efficiency for massive data streams during the computing phase. Finally, we test and compare the two parallel scheduling strategies on a large quantity of synthetic and real stream datas. The simulation experiments and compuatation results in practice both prove the accuracy of analysis on performance and power efficiency.

Sign in / Sign up

Export Citation Format

Share Document