Processing Data Streams

2020 ◽  
pp. 20-39
Author(s):  
Parimala N.
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Data Streams ◽  
Data Stream ◽  
Continuous Queries ◽  
Stream Data ◽  
Continuous Sequence ◽  
Static Data ◽  
Transactional Data ◽  
Events Data

A data stream is a real-time continuous sequence that may be comprised of data or events. Data stream processing is different from static data processing which resides in a database. The data stream data is seen only once. It is too voluminous to store statically. A small portion of data called a window is considered at a time for querying, computing aggregates, etc. In this chapter, the authors explain the different types of window movement over incoming data. A query on a stream is repeatedly executed on the new data created by the movement of the window. SQL extensions to handle continuous queries is addressed in this chapter. Streams that contain transactional data as well as those that contain events are considered.

SBASH Stack Based Allocation of Sheer Window Architecture for Real Time Stream Data Processing

2020 ◽  
Vol 1 (1) ◽  
pp. 1-21
Author(s):  
Devesh Kumar Lal ◽  
Ugrasen Suman
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Data Streams ◽  
Time Complexity ◽  
Data Stream ◽  
Stream Data ◽  
Time Data ◽  
Processing Data ◽  
Stream Data Processing ◽  
Window Selection

The processing of real-time data streams is complex with large number of volume and variety. The volume and variety of data streams enhances a number of processing units to run in real time. The required number of processing units used for processing data streams are lowered by using a windowing mechanism. Therefore, the appropriate size of window selection is vital for stream data processing. The coarse size window will directly affect the overall processing time. On the other hand, a finely sized window has to deal with an increased number of management costs. In order to manage such streams of data, we have proposed a SBASH architecture, which can be helpful for determining a unipartite size of a sheer window. The sheer window reduces the overall latency of data stream processing by a certain extent. The time complexity to process such sheer window is equivalent to w log n w. These windows are allocated and retrieved in a stack-based manner, where stacks ≥ n, which is helpful in reducing the number of comparisons made during retrieval.

scholarly journals Measuring the Effectiveness of Adaptive Random Forest for Handling Concept Drift in Big Data Streams

Entropy ◽  
2021 ◽  
Vol 23 (7) ◽  
pp. 859
Author(s):  
Abdulaziz O. AlQabbany ◽  
Aqil M. Azmi
Keyword(s):  
Big Data ◽  
Random Forest ◽  
Real Time ◽  
Data Streams ◽  
Learning Algorithm ◽  
Concept Drift ◽  
The United States ◽  
Careful Consideration ◽  
Data Sets ◽  
Stream Data

We are living in the age of big data, a majority of which is stream data. The real-time processing of this data requires careful consideration from different perspectives. Concept drift is a change in the data’s underlying distribution, a significant issue, especially when learning from data streams. It requires learners to be adaptive to dynamic changes. Random forest is an ensemble approach that is widely used in classical non-streaming settings of machine learning applications. At the same time, the Adaptive Random Forest (ARF) is a stream learning algorithm that showed promising results in terms of its accuracy and ability to deal with various types of drift. The incoming instances’ continuity allows for their binomial distribution to be approximated to a Poisson(1) distribution. In this study, we propose a mechanism to increase such streaming algorithms’ efficiency by focusing on resampling. Our measure, resampling effectiveness (ρ), fuses the two most essential aspects in online learning; accuracy and execution time. We use six different synthetic data sets, each having a different type of drift, to empirically select the parameter λ of the Poisson distribution that yields the best value for ρ. By comparing the standard ARF with its tuned variations, we show that ARF performance can be enhanced by tackling this important aspect. Finally, we present three case studies from different contexts to test our proposed enhancement method and demonstrate its effectiveness in processing large data sets: (a) Amazon customer reviews (written in English), (b) hotel reviews (in Arabic), and (c) real-time aspect-based sentiment analysis of COVID-19-related tweets in the United States during April 2020. Results indicate that our proposed method of enhancement exhibited considerable improvement in most of the situations.

Knowledge Discovery From Evolving Data Streams

2019 ◽  
pp. 19-39
Author(s):  
Prasanna Lakshmi Kompalli
Keyword(s):  
Real Time ◽  
Data Streams ◽  
Data Stream ◽  
Concept Drift ◽  
Data Stream Mining ◽  
Time Data ◽  
Stream Mining ◽  
New Challenges ◽  
Mining Data Streams ◽  
Different Sources

Data coming from different sources is referred to as data streams. Data stream mining is an online learning technique where each data point must be processed as the data arrives and discarded as the processing is completed. Progress of technologies has resulted in the monitoring these data streams in real time. Data streams has created many new challenges to the researchers in real time. The main features of this type of data are they are fast flowing, large amounts of data which are continuous and growing in nature, and characteristics of data might change in course of time which is termed as concept drift. This chapter addresses the problems in mining data streams with concept drift. Due to which, isolating the correct literature would be a grueling task for researchers and practitioners. This chapter tries to provide a solution as it would be an amalgamation of all techniques used for data stream mining with concept drift.

scholarly journals REAL-TIME EARTHQUAKE MONITORING WITH SPATIO-TEMPORAL FIELDS

2017 ◽  
Vol IV-4/W2 ◽  
pp. 215-222 ◽  
Author(s):  
J. C. Whittier ◽  
S. Nittel ◽  
I. Subasinghe
Keyword(s):  
Real Time ◽  
Data Streams ◽  
Data Stream ◽  
Apache Spark ◽  
Sensor Data ◽  
Live Streaming ◽  
Earthquake Activity ◽  
Maximum Displacement ◽  
Earthquake Event ◽  
Spatio Temporal

With live streaming sensors and sensor networks, increasingly large numbers of individual sensors are deployed in physical space. Sensor data streams are a fundamentally novel mechanism to deliver observations to information systems. They enable us to represent spatio-temporal continuous phenomena such as radiation accidents, toxic plumes, or earthquakes almost as instantaneously as they happen in the real world. Sensor data streams discretely sample an earthquake, while the earthquake is continuous over space and time. Programmers attempting to integrate many streams to analyze earthquake activity and scope need to write code to integrate potentially very large sets of asynchronously sampled, concurrent streams in tedious application code. In previous work, we proposed the field stream data model (Liang et al., 2016) for data stream engines. Abstracting the stream of an individual sensor as a temporal field, the field represents the Earth’s movement at the sensor position as continuous. This simplifies analysis across many sensors significantly. In this paper, we undertake a feasibility study of using the field stream model and the open source Data Stream Engine (DSE) Apache Spark(Apache Spark, 2017) to implement a real-time earthquake event detection with a subset of the 250 GPS sensor data streams of the Southern California Integrated GPS Network (SCIGN). The field-based real-time stream queries compute maximum displacement values over the latest query window of each stream, and related spatially neighboring streams to identify earthquake events and their extent. Further, we correlated the detected events with an USGS earthquake event feed. The query results are visualized in real-time.

scholarly journals Efficient Processing of ContinuousSkylineQuery over Smarter Traffic Data Stream for Cloud Computing

2013 ◽  
Vol 2013 ◽  
pp. 1-10 ◽  
Author(s):  
Wang Hanning ◽  
Xu Weixiang ◽  
Jiulin Yang ◽  
Lili Wei ◽  
Jia Chaolong
Keyword(s):  
Cloud Computing ◽  
Real Time ◽  
Data Stream ◽  
Sliding Window ◽  
Continuous Query ◽  
Traffic Data ◽  
Stream Data ◽  
Smart Transportation ◽  
Efficient Processing ◽  
Query Algorithm

The analyzing and processing of multisource real-time transportation data stream lay a foundation for the smart transportation's sensibility, interconnection, integration, and real-time decision making. Strong computing ability and valid mass data management mode provided by the cloud computing, is feasible for handlingSkylinecontinuous query in the mass distributed uncertain transportation data stream. In this paper, we gave architecture of layered smart transportation about data processing, and we formalized the description about continuous query over smart transportation dataSkyline. Besides, we proposedmMR-SUDSalgorithm (Skylinequery algorithm of uncertain transportation stream data based onmicro-batchinMap Reduce) based on sliding window division and architecture.

scholarly journals MCDAStream: a real-time data stream clustering based on micro-cluster density and attraction

2018 ◽  
Vol 7 (2) ◽  
pp. 270 ◽  
Author(s):  
Shyam Sunder Reddy K ◽  
Shoba Bindu C
Keyword(s):  
Real Time ◽  
Data Streams ◽  
Data Stream ◽  
Clustering Algorithm ◽  
Positional Information ◽  
Time Data ◽  
Stream Clustering ◽  
Real Time Data ◽  
Cluster Density ◽  
Data Stream Clustering

Real-time data stream clustering has been widely used in many fields, and it can extract useful information from massive sets of data. Most of the existing density-based algorithms cluster the data streams based on the density within the micro-clusters. These algorithms completely omit the data density in the area between the micro-clusters and recluster the micro-clusters based on erroneous assumptions about the distribution of the data within and between the micro-clusters that lead to poor clustering results. This paper describes a novel density-based clustering algorithm for evolving data streams called MCDAStream, which clusters the data stream based on micro-cluster density and attraction between the micro-clusters. The attraction of micro-clusters characterizes the positional information of the data points in each micro-cluster. We generate better clustering results by considering both micro-cluster density and attraction of micro-clusters. The quality of the proposed algorithm is evaluated on various synthetic and real-time datasets with distinct characteristics and quality metrics.

scholarly journals CHANGE SEMANTIC CONSTRAINED ONLINE DATA CLEANING METHOD FOR REAL-TIME OBSERVATIONAL DATA STREAM

2016 ◽  
Vol XLI-B2 ◽  
pp. 177-183
Author(s):  
Yulin Ding ◽  
Hui Lin ◽  
Rongrong Li
Keyword(s):  
Real Time ◽  
Observational Data ◽  
Data Streams ◽  
Data Stream ◽  
Data Cleaning ◽  
Data Distribution ◽  
Online Data ◽  
Filter Parameter ◽  
Changing Patterns ◽  
Cleaning Methods

Recent breakthroughs in sensor networks have made it possible to collect and assemble increasing amounts of real-time observational data by observing dynamic phenomena at previously impossible time and space scales. Real-time observational data streams present potentially profound opportunities for real-time applications in disaster mitigation and emergency response, by providing accurate and timeliness estimates of environment’s status. However, the data are always subject to inevitable anomalies (including errors and anomalous changes/events) caused by various effects produced by the environment they are monitoring. The “big but dirty” real-time observational data streams can rarely achieve their full potential in the following real-time models or applications due to the low data quality. Therefore, timely and meaningful online data cleaning is a necessary pre-requisite step to ensure the quality, reliability, and timeliness of the real-time observational data. <br><br> In general, a straightforward streaming data cleaning approach, is to define various types of models/classifiers representing normal behavior of sensor data streams and then declare any deviation from this model as normal or erroneous data. The effectiveness of these models is affected by dynamic changes of deployed environments. Due to the changing nature of the complicated process being observed, real-time observational data is characterized by diversity and dynamic, showing a typical Big (Geo) Data characters. Dynamics and diversity is not only reflected in the data values, but also reflected in the complicated changing patterns of the data distributions. This means the pattern of the real-time observational data distribution is not <i>stationary or static</i> but <i>changing and dynamic</i>. After the data pattern changed, it is necessary to adapt the model over time to cope with the changing patterns of real-time data streams. Otherwise, the model will not fit the following observational data streams, which may led to large estimation error. In order to achieve the best generalization error, it is an important challenge for the data cleaning methodology to be able to characterize the behavior of data stream distributions and adaptively update a model to include new information and remove old information. However, the complicated data changing property invalidates traditional data cleaning methods, which rely on the assumption of a stationary data distribution, and drives the need for more dynamic and adaptive online data cleaning methods. <br><br> To overcome these shortcomings, this paper presents a change semantics constrained online filtering method for real-time observational data. Based on the principle that the filter parameter should vary in accordance to the data change patterns, this paper embeds semantic description, which quantitatively depicts the change patterns in the data distribution to self-adapt the filter parameter automatically. Real-time observational water level data streams of different precipitation scenarios are selected for testing. Experimental results prove that by means of this method, more accurate and reliable water level information can be available, which is prior to scientific and prompt flood assessment and decision-making.

Sign in / Sign up

Export Citation Format

Share Document