Research on real-time data centre reconstruction technology based on big data technology

2017 ◽  
pp. 203-206
Author(s):  
Li Xianhui ◽  
Sheng Zhou ◽  
Shengpeng Ji ◽  
Gang Zhen ◽  
Yang He ◽  
...  
Keyword(s):  
Big Data ◽  
Real Time ◽  
Time Data ◽  
Data Centre ◽  
Real Time Data ◽  
Big Data Technology

scholarly journals Study on real-time prediction model of railway passenger flow based on big data technology

2022 ◽  
Vol 355 ◽  
pp. 02025
Author(s):  
Yiyi Yin ◽  
Yong Zhang ◽  
Zhengzheng Wei ◽  
Xiang Zhao
Keyword(s):  
Big Data ◽  
Prediction Model ◽  
Real Time ◽  
Time Data ◽  
Message Queue ◽  
Passenger Flow ◽  
Time Prediction ◽  
Real Time Data ◽  
Railway Passenger ◽  
Big Data Technology

In order to solve the limitation of traditional offline forecasting application scenarios, the author uses a variety of big data open source frameworks and tools to combine with railway real-time data, and proposes a real-time prediction model of railway passenger flow. The model architecture is divided into four levels from bottom to top: data source layer, data transmission layer, prediction calculation layer and application layer. The main components of the model are data flow and prediction flow. Through message queue and ETL, the data process part realizes the synchronization of offline data and real-time data; through the big data technology frameworks such as Spark, Redis and Hive and the GBDT (Gradient Boosting Tree) algorithm, the prediction process partially realizes the real-time passenger flow of the train OD section prediction. The experimental results show that the model proposed by the author has certain practicability and accuracy both in performance and prediction accuracy.

Big Data Management in the Context of Real-Time Data Warehousing

2013 ◽  
pp. 150-176
Author(s):  
M. Asif Naeem ◽  
Gillian Dobbie ◽  
Gerald Weber
Keyword(s):  
Big Data ◽  
Data Integration ◽  
Real Time ◽  
Real Life ◽  
Skewed Distribution ◽  
Stream Data ◽  
Time Data ◽  
Master Data ◽  
Real Time Data ◽  
Resource Aware

In order to make timely and effective decisions, businesses need the latest information from big data warehouse repositories. To keep these repositories up to date, real-time data integration is required. An important phase in real-time data integration is data transformation where a stream of updates, which is huge in volume and infinite, is joined with large disk-based master data. Stream processing is an important concept in Big Data, since large volumes of data are often best processed immediately. A well-known algorithm called Mesh Join (MESHJOIN) was proposed to process stream data with disk-based master data, which uses limited memory. MESHJOIN is a candidate for a resource-aware system setup. The problem that the authors consider in this chapter is that MESHJOIN is not very selective. In particular, the performance of the algorithm is always inversely proportional to the size of the master data table. As a consequence, the resource consumption is in some scenarios suboptimal. They present an algorithm called Cache Join (CACHEJOIN), which performs asymptotically at least as well as MESHJOIN but performs better in realistic scenarios, particularly if parts of the master data are used with different frequencies. In order to quantify the performance differences, the authors compare both algorithms with a synthetic dataset of a known skewed distribution as well as TPC-H and real-life datasets.

Performance Improvement IoT Applications Through Multimedia Analytics Using Big Data Stream Computing Platforms

2018 ◽  
pp. 200-221
Author(s):  
Rizwan Patan ◽  
Rajasekhara Babu M ◽  
Suresh Kallam
Keyword(s):  
Big Data ◽  
Real Time ◽  
Performance Improvement ◽  
Data Stream ◽  
Real Data ◽  
Stream Computing ◽  
Time Data ◽  
Real Time Data ◽  
Computing Platforms ◽  
Time And Energy

A Big Data Stream Computing (BDSC) Platform handles real-time data from various applications such as risk management, marketing management and business intelligence. Now a days Internet of Things (IoT) deployment is increasing massively in all the areas. These IoTs engender real-time data for analysis. Existing BDSC is inefficient to handle Real-data stream from IoTs because the data stream from IoTs is unstructured and has inconstant velocity. So, it is challenging to handle such real-time data stream. This work proposes a framework that handles real-time data stream through device control techniques to improve the performance. The frame work includes three layers. First layer deals with Big Data platforms that handles real data streams based on area of importance. Second layer is performance layer which deals with performance issues such as low response time, and energy efficiency. The third layer is meant for Applying developed method on existing BDSC platform. The experimental results have been shown a performance improvement 20%-30% for real time data stream from IoT application.

Trends and Technologies in Big Data Processing

2020 ◽  
pp. 17-42
Author(s):  
Amitava Choudhury ◽  
Kalpana Rangra
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Internet Of Things ◽  
Data Processing ◽  
Real Time ◽  
Computing Technology ◽  
Time Data ◽  
Big Data Processing ◽  
Real Time Data ◽  
Real Time Data Processing

Data type and amount in human society is growing at an amazing speed, which is caused by emerging new services such as cloud computing, internet of things, and location-based services. The era of big data has arrived. As data has been a fundamental resource, how to manage and utilize big data better has attracted much attention. Especially with the development of the internet of things, how to process a large amount of real-time data has become a great challenge in research and applications. Recently, cloud computing technology has attracted much attention to high performance, but how to use cloud computing technology for large-scale real-time data processing has not been studied. In this chapter, various big data processing techniques are discussed.

Customer behavior analysis using real-time data processing

2019 ◽  
Vol 31 (1) ◽  
pp. 265-290 ◽  
Author(s):  
Ganjar Alfian ◽  
Muhammad Fazal Ijaz ◽  
Muhammad Syafrudin ◽  
M. Alex Syaekhoni ◽  
Norma Latif Fitriyani ◽  
...  
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Association Rule ◽  
Customer Behavior ◽  
Time Data ◽  
Content Type ◽  
The Real ◽  
Real Time Data ◽  
Real Time Data Processing ◽  
Big Data Technology

PurposeThe purpose of this paper is to propose customer behavior analysis based on real-time data processing and association rule for digital signage-based online store (DSOS). The real-time data processing based on big data technology (such as NoSQL MongoDB and Apache Kafka) is utilized to handle the vast amount of customer behavior data.Design/methodology/approachIn order to extract customer behavior patterns, customers’ browsing history and transactional data from digital signage (DS) could be used as the input for decision making. First, the authors developed a DSOS and installed it in different locations, so that customers could have the experience of browsing and buying a product. Second, the real-time data processing system gathered customers’ browsing history and transaction data as it occurred. In addition, the authors utilized the association rule to extract useful information from customer behavior, so it may be used by the managers to efficiently enhance the service quality.FindingsFirst, as the number of customers and DS increases, the proposed system was capable of processing a gigantic amount of input data conveniently. Second, the data set showed that as the number of visit and shopping duration increases, the chance of products being purchased also increased. Third, by combining purchasing and browsing data from customers, the association rules from the frequent transaction pattern were achieved. Thus, the products will have a high possibility to be purchased if they are used as recommendations.Research limitations/implicationsThis research empirically supports the theory of association rule that frequent patterns, correlations or causal relationship found in various kinds of databases. The scope of the present study is limited to DSOS, although the findings can be interpreted and generalized in a global business scenario.Practical implicationsThe proposed system is expected to help management in taking decisions such as improving the layout of the DS and providing better product suggestions to the customer.Social implicationsThe proposed system may be utilized to promote green products to the customer, having a positive impact on sustainability.Originality/valueThe key novelty of the present study lies in system development based on big data technology to handle the enormous amounts of data as well as analyzing the customer behavior in real time in the DSOS. The real-time data processing based on big data technology (such as NoSQL MongoDB and Apache Kafka) is used to handle the vast amount of customer behavior data. In addition, the present study proposed association rule to extract useful information from customer behavior. These results can be used for promotion as well as relevant product recommendations to DSOS customers. Besides in today’s changing retail environment, analyzing the customer behavior in real time in DSOS helps to attract and retain customers more efficiently and effectively, and retailers can get a competitive advantage over their competitors.

scholarly journals Urban Planning and Smart City Decision Management Empowered by Real-Time Data Processing Using Big Data Analytics

Sensors ◽  
2018 ◽  
Vol 18 (9) ◽  
pp. 2994 ◽  
Author(s):  
Bhagya Silva ◽  
Murad Khan ◽  
Changsu Jung ◽  
Jihun Seo ◽  
Diyan Muhammad ◽  
...  
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Real Time ◽  
Smart City ◽  
Data Analytics ◽  
Smart Cities ◽  
Big Data Analytics ◽  
Time Data ◽  
Real Time Data

The Internet of Things (IoT), inspired by the tremendous growth of connected heterogeneous devices, has pioneered the notion of smart city. Various components, i.e., smart transportation, smart community, smart healthcare, smart grid, etc. which are integrated within smart city architecture aims to enrich the quality of life (QoL) of urban citizens. However, real-time processing requirements and exponential data growth withhold smart city realization. Therefore, herein we propose a Big Data analytics (BDA)-embedded experimental architecture for smart cities. Two major aspects are served by the BDA-embedded smart city. Firstly, it facilitates exploitation of urban Big Data (UBD) in planning, designing, and maintaining smart cities. Secondly, it occupies BDA to manage and process voluminous UBD to enhance the quality of urban services. Three tiers of the proposed architecture are liable for data aggregation, real-time data management, and service provisioning. Moreover, offline and online data processing tasks are further expedited by integrating data normalizing and data filtering techniques to the proposed work. By analyzing authenticated datasets, we obtained the threshold values required for urban planning and city operation management. Performance metrics in terms of online and offline data processing for the proposed dual-node Hadoop cluster is obtained using aforementioned authentic datasets. Throughput and processing time analysis performed with regard to existing works guarantee the performance superiority of the proposed work. Hence, we can claim the applicability and reliability of implementing proposed BDA-embedded smart city architecture in the real world.

Sign in / Sign up

Export Citation Format

Share Document