Real-time data pre-processing technique for efficient feature extraction in large scale datasets

2008 ◽  
Author(s):  
Ying Liu ◽  
Lucian V. Lita ◽  
R. Stefan Niculescu ◽  
Kun Bai ◽  
Prasenjit Mitra ◽  
...  
Keyword(s):  
Feature Extraction ◽  
Real Time ◽  
Large Scale ◽  
Processing Technique ◽  
Time Data ◽  
Real Time Data

scholarly journals An LSTM-Based Method Considering History and Real-Time Data for Passenger Flow Prediction

2020 ◽  
Vol 10 (11) ◽  
pp. 3788 ◽  
Author(s):  
Qi Ouyang ◽  
Yongbo Lv ◽  
Jihui Ma ◽  
Jing Li
Keyword(s):  
Feature Extraction ◽  
Real Time ◽  
Short Term Memory ◽  
Historical Data ◽  
Time Interval ◽  
Information Coding ◽  
Time Data ◽  
Passenger Flow ◽  
Flow Prediction ◽  
Real Time Data

With the development of big data and deep learning, bus passenger flow prediction considering real-time data becomes possible. Real-time traffic flow prediction helps to grasp real-time passenger flow dynamics, provide early warning for a sudden passenger flow and data support for real-time bus plan changes, and improve the stability of urban transportation systems. To solve the problem of passenger flow prediction considering real-time data, this paper proposes a novel passenger flow prediction network model based on long short-term memory (LSTM) networks. The model includes four parts: feature extraction based on Xgboost model, information coding based on historical data, information coding based on real-time data, and decoding based on a multi-layer neural network. In the feature extraction part, the data dimension is increased by fusing bus data and points of interest to improve the number of parameters and model accuracy. In the historical information coding part, we use the date as the index in the LSTM structure to encode historical data and provide relevant information for prediction; in the real-time data coding part, the daily half-hour time interval is used as the index to encode real-time data and provide real-time prediction information; in the decoding part, the passenger flow data for the next two 30 min interval outputs by decoding all the information. To our best knowledge, it is the first time to real-time information has been taken into consideration in passenger flow prediction based on LSTM. The proposed model can achieve better accuracy compared to the LSTM and other baseline methods.

A distributed real-time data prediction framework for large-scale time-series data using stream processing

2017 ◽  
Vol 10 (2) ◽  
pp. 145-165 ◽  
Author(s):  
Kehe Wu ◽  
Yayun Zhu ◽  
Quan Li ◽  
Ziwei Wu
Keyword(s):  
Time Series ◽  
Real Time ◽  
Large Scale ◽  
Time Series Data ◽  
Data Sources ◽  
Series Data ◽  
Time Data ◽  
Content Type ◽  
Data Prediction ◽  
Real Time Data

Purpose The purpose of this paper is to propose a data prediction framework for scenarios which require forecasting demand for large-scale data sources, e.g., sensor networks, securities exchange, electric power secondary system, etc. Concretely, the proposed framework should handle several difficult requirements including the management of gigantic data sources, the need for a fast self-adaptive algorithm, the relatively accurate prediction of multiple time series, and the real-time demand. Design/methodology/approach First, the autoregressive integrated moving average-based prediction algorithm is introduced. Second, the processing framework is designed, which includes a time-series data storage model based on the HBase, and a real-time distributed prediction platform based on Storm. Then, the work principle of this platform is described. Finally, a proof-of-concept testbed is illustrated to verify the proposed framework. Findings Several tests based on Power Grid monitoring data are provided for the proposed framework. The experimental results indicate that prediction data are basically consistent with actual data, processing efficiency is relatively high, and resources consumption is reasonable. Originality/value This paper provides a distributed real-time data prediction framework for large-scale time-series data, which can exactly achieve the requirement of the effective management, prediction efficiency, accuracy, and high concurrency for massive data sources.

Technology, Nature, Software and Networks: Materializing the Post-Romantic Landscape

Leonardo ◽  
2011 ◽  
Vol 44 (2) ◽  
pp. 101-106 ◽  
Author(s):  
Chris Welsby
Keyword(s):  
New Media ◽  
Real Time ◽  
Large Scale ◽  
Natural World ◽  
Time Data ◽  
Real Time Data ◽  
Weather Stations ◽  
Selection Of ◽  
Made In ◽  
Media Installation

The author began making films and installations in the early 1970s. Although he has worked across a range of media, he has always concentrated on one particular theme that he conceptualizes as a two-sided question: How do we see ourselves in relation to the natural world, and how should we position our selves and our technologies within it? This essay traces some of the threads of these seminal ideas through a selection of works made in the years between 1974 and 2009. The author concludes with a detailed description of Tree Studies, a large-scale new media installation powered by real-time data from weather stations around the planet.

INS real-time data processing composite system using large scale general purpose computers

1983 ◽  
Vol 213 (2-3) ◽  
pp. 317-327 ◽  
Author(s):  
Jun Kokame ◽  
Motonobu Takano ◽  
Tomoko Oshikubo ◽  
Kurazo Chiba ◽  
Kumataro Ukai ◽  
...  
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Large Scale ◽  
Composite System ◽  
General Purpose ◽  
Time Data ◽  
Real Time Data ◽  
Real Time Data Processing

scholarly journals Impact of Fog and Cloud Computing Applications to Reserve a Seat in Railways

2020 ◽  
Vol 13 (1) ◽  
pp. 28
Author(s):  
Gajender Kumar Kumar
Keyword(s):  
Real Time ◽  
Large Scale ◽  
Real Data ◽  
Third Parties ◽  
College Level ◽  
Time Data ◽  
Data Pipeline ◽  
Real Time Data ◽  
Multiple Devices ◽  
Sensor Cloud

As India's largest employer has gone digital, this is the beginning of mass acquisition in a new dimension. Indian Railways serves on a peripheral scale in the size of each aircraft. It carries 23,000 passengers and over 3 million tonnes daily, with more than 12,000 passengers traveling at 115,000 km and 7000 stations. But as the nation's largest employer has begun to digitalize its services operations, it is beginning to gain mass in an emerging dimension. The network collects around 100 datasets per user (one truffle is enough to store 500–1000 movies). This passenger booking plate has 25 million users, leading to approximately 800,000 daily transactions. Such is the creation of a digital scale railway ecosystem, a fertile ground for entrepreneurs, employers and large scale technology for testing and testing. And while Railways has made a good start by integrating third parties with third parties, the company can create a seamless experience in booking, payment and other services at the top of the railway data pipeline. Here we have several ways to verify a passenger or passenger ticket, which will be discussed one by one on Fig.1. We understand here that the N computer system (server) is available in the availability center and the availability train provided by the NECC (North Eastern Carrying Corporation Ltd). Each seat or chair acts as a spring (Smart) chair, which means that we are connected to a ticket scanner or chair that can scan tickets and send the actual data to the field node (fog node). Later, the latest analysis will result in the proposed data and the final data that will be sent to the cloud. We are connected to multiple devices to share real-time data and updated real-time data on the cloud. We have created a fog environment and a cloud environment. We have implemented this with the help of clipping and sending data by sending real data separately. We have created a smart gateway between military (Servers) and customer (sensor), cloud and customer (sensor). In this paper, there is the solution that can enhanced the chances of getting confirm ticket, based on individual server load in coaches of the train, according to the increase in the number of sensors request, in the context of processing and considering three cases , it is an answer by analysing cloud-fog scenario. Up to this implementation the simple round robin algorithm is also used solution for load balancing purpose. The method is implemented by using a class student attendance system in a college level.

Sign in / Sign up

Export Citation Format

Share Document