scholarly journals Data Pipeline Architecture with Near Real-Time Streaming Multiple Source Indonesian Online News Data Lake

2020 ◽  
Vol 3 (1) ◽  
Author(s):  
Angelina Pramana Thenata
Keyword(s):  
Real Time ◽  
Architectural Design ◽  
Rapid Development ◽  
Online News ◽  
Pipeline Architecture ◽  
Time Data ◽  
Literature Study ◽  
Data Pipeline ◽  
Data Volume ◽  
The Government

The rapid development of information has made online news increasingly needed. Online news attracts readers' attention by providing convenience and speed in presenting news from various fields. However, the large amount (volume) of online news that spreads in a short time (velocity) and the public's need to consume news in various references (variety) can affect people's lives. Therefore, the government as the regulator and news agencies need to monitor online news circulating. Based on these problems, the researcher proposes a data lake architectural design that is suitable for online news and can run in real-time. Data lakes can solve the main problems of Big Data (volume, velocity, variety). In proposing this data lake architecture, the researcher conducted a literature study and analyzed the flow of the data lake architecture according to online news. Furthermore, the researcher will use this architecture to combine and uniform the online news data structure from several online news channels and then stream it in real-time to fill the data lake. The results of using the data lake architecture for online news will be stored on MongoDB which functions as a database to store all data for both the short and long term. Finally, this data lake will be a means to accommodate, dive into, and analyze the circulating online news data. Keywords – Data Lake, Online News, Real-Time.

A Real-Time Fiber Optical System for Wellbore Monitoring: A Johan Sverdrup Case Study

2021 ◽  
Author(s):  
Maximilian Georg Schuberth ◽  
Håkon Sunde Bakka ◽  
Claire Emma Birnie ◽  
Stefan Dümmong ◽  
Kjetil Eik Haavik ◽  
...  
Keyword(s):  
Decision Making ◽  
Real Time ◽  
Large Data ◽  
Pipeline Architecture ◽  
Distributed Temperature Sensing ◽  
Time Data ◽  
Data Pipeline ◽  
Start Up ◽  
Custom Made ◽  
Gas Lift

Abstract Fiber Optic (FO) sensing capabilities for downhole monitoring include, among other techniques, Distributed Temperature Sensing (DTS) and Distributed Acoustic Sensing (DAS). The appeal of DTS and DAS data is based on its high temporal and spatial sampling, allowing for very fine localization of processes in a wellbore. Furthermore, the broad frequency spectrum that especially DAS data is acquired with, enables observations, ranging from more continuous effects like oil flow, to more distinct effects like opening and closing of valves. Due to the high data volume of hundreds of Gb per well per hour, DAS data has traditionally been acquired acquisition-based, where data is recorded for a limited amount of time and processed at a later point in time. This limits the decision-making capability based on this data as reacting to events is only possible long after the event occurred. Equinor has addressed these decision-making shortcomings by building a real-time streaming solution for transferring, processing, and interpretation of its FO data at the Johan Sverdrup field in the North Sea. The streaming solution for FO data consists of offshore interrogators streaming raw DAS and DTS data via a dedicated bandwidth to an onshore processing cluster. There, DAS data is transformed into FO feature data, e.g., Frequency Band Energies, which are heavily decimated versions of the raw data; allowing insight extraction, while significantly reducing data volumes. DTS and DAS FO feature data are then streamed to a custom-made, cloud-based visualization and integration platform. This cloud-based platform allows efficient inspection of large data sets, control and evaluation of applications based on these data, and sharing of FO data within the Johan Sverdrup asset. During the last year, this FO data streaming pipeline has processed several tens of PB of FO data, monitoring a range of well operations and processes. Qualitatively, the benefits and potential of the real-time data acquisitions have been illustrated by providing a greater understanding of current well conditions and processes. Alongside the FO data pipeline, multiple prototype applications have been developed for automated monitoring of Gas Lift Valves, Safety Valve operations, Gas Lift rate estimation, and monitoring production start-up, all providing insights in real-time. For certain use cases, such as monitoring production start-up, the FO data provides a previously non-existent monitoring solution. In this paper, we will discuss in detail the FO data pipeline architecture from-platform-to-cloud, illustrate several data examples, and discuss the way-forward for "real-time" FO data analytics.

scholarly journals A Comprehensive Survey on Cyber-Physical Smart Grid Testbed Architectures: Requirements and Challenges

Electronics ◽  
2021 ◽  
Vol 10 (9) ◽  
pp. 1043
Author(s):  
Abdallah A. Smadi ◽  
Babatunde Tobi Ajao ◽  
Brian K. Johnson ◽  
Hangtian Lei ◽  
Yacine Chakhchoukh ◽  
...  
Keyword(s):  
Smart Grid ◽  
Real Time ◽  
Smart Grids ◽  
Data Exchange ◽  
Information Technologies ◽  
New Technologies ◽  
Rapid Development ◽  
Environmental Benefits ◽  
Future Research ◽  
Time Data

The integration of improved control techniques with advanced information technologies enables the rapid development of smart grids. The necessity of having an efficient, reliable, and flexible communication infrastructure is achieved by enabling real-time data exchange between numerous intelligent and traditional electrical grid elements. The performance and efficiency of the power grid are enhanced with the incorporation of communication networks, intelligent automation, advanced sensors, and information technologies. Although smart grid technologies bring about valuable economic, social, and environmental benefits, testing the combination of heterogeneous and co-existing Cyber-Physical-Smart Grids (CP-SGs) with conventional technologies presents many challenges. The examination for both hardware and software components of the Smart Grid (SG) system is essential prior to the deployment in real-time systems. This can take place by developing a prototype to mimic the real operational circumstances with adequate configurations and precision. Therefore, it is essential to summarize state-of-the-art technologies of industrial control system testbeds and evaluate new technologies and vulnerabilities with the motivation of stimulating discoveries and designs. In this paper, a comprehensive review of the advancement of CP-SGs with their corresponding testbeds including diverse testing paradigms has been performed. In particular, we broadly discuss CP-SG testbed architectures along with the associated functions and main vulnerabilities. The testbed requirements, constraints, and applications are also discussed. Finally, the trends and future research directions are highlighted and specified.

Data Reduction Techniques for Near Real-Time Decision Making in Fall Prediction Systems

2018 ◽  
pp. 52-64
Author(s):  
Masoud Hemmatpour ◽  
Renato Ferrero ◽  
Filippo Gandino ◽  
Bartolomeo Montrucchio ◽  
Maurizio Rebaudengo
Keyword(s):  
Real Time ◽  
High Speed ◽  
Time Data ◽  
Reduction Techniques ◽  
Fall Prediction ◽  
Real Time Data ◽  
Data Volume ◽  
Prediction Systems ◽  
Unintentional Falls ◽  
Health Service Costs

Unintentional falls are a frequent cause of hospitalization that mostly increases health service costs due to injuries. Fall prediction systems strive to reduce injuries and provide fast help to the users. Typically, such systems collect data continuously at a high speed through a device directly attached to the user. Whereas such systems are implemented in devices with limited resources, data volume is significantly important. In this chapter, a real-time data analyzer and reducer is proposed in order to manage the data volume of fall prediction systems.

scholarly journals Research on Security Framework of Mobile Application in Dispatch and Control Domain of Power Grid

2018 ◽  
Vol 173 ◽  
pp. 02029
Author(s):  
XU Jiahui ◽  
YU Hongyuan ◽  
WANG Gang ◽  
WANG Zi ◽  
BAI Jingjie ◽  
...  
Keyword(s):  
Real Time ◽  
Mobile Application ◽  
Power Grid ◽  
Rapid Development ◽  
Mobile Internet ◽  
Internet Technology ◽  
Security Framework ◽  
Time Data ◽  
Real Time Data ◽  
And Control

The rapid development of mobile Internet technology and the wide spread of smart terminals have brought opportunities for the transformation of power grid business model. Compared to the non-real-time information, the real-time and running data of dispatch and control domain is easy to be intercepted and cracked. To solve this problem, this paper presents a new approach to mobile application security framework for the power grid control field. It is to realize secondary encryption by using the method of MD5+AES mixed encryption algorithm and combining the time stamp in real-time data transmission process. At the same time it is to prevent cross-border operations and brute force by using Token authentication and Session technology. China EPRI safety test results show that the application of the framework significantly improves the integrity, safety and reliability of real-time data in power grid control.

The Design of Real-Time and High-Speed Video Processing Platform for Aerial UAV Based on DM642

2014 ◽  
Vol 1061-1062 ◽  
pp. 1186-1189
Author(s):  
Ming Zhe Wei ◽  
Wan Wei Tang
Keyword(s):  
Real Time ◽  
Video Processing ◽  
High Speed ◽  
Rapid Development ◽  
Processing System ◽  
Video Signal ◽  
Good Effect ◽  
Time Data ◽  
Spatial Interference ◽  
On Chip

With the rapid development of aerial UAV (Unmanned Aerial Vehicle), the design of real-time data acquisition and transmission system for the video signal has a new applied field. It is different from traditional video acquisition and processing system, aerial video signal has the problems of screen jitter and spatial interference. The processing algorithm of aerial UAV airborne video signal is put forward in the paper, and the platform of high speed procession is constructed based on chip TMS320DM642, and get a good effect.

Design of Extended Real-time Data Pipeline System Architecture

2015 ◽  
Vol 42 (8) ◽  
pp. 1010-1021
Author(s):  
Hoseung Shin ◽  
Sungwon Kang ◽  
Jihyun Lee
Keyword(s):  
Real Time ◽  
System Architecture ◽  
Pipeline System ◽  
Time Data ◽  
Data Pipeline ◽  
Real Time Data

scholarly journals IoT Based Vehicle Mounted Weight Sensors

2019 ◽  
Vol 8 (4) ◽  
pp. 5686-5691
Keyword(s):  
Internet Of Things ◽  
Real Time ◽  
Central Government ◽  
Smart Sensors ◽  
Time Data ◽  
The Real ◽  
Weight Variation ◽  
Block Chain ◽  
The Government ◽  
The Internet Of Things

The as most of the goods trucks in the developing nation are overloaded with goods, and this needs to be controlled via smart way. This can be achieved using the Smart Sensors and Internet of Things (IoT), as these are the emerging technologies. In these paper we propose the weight mounted sensors can dynamical give an accurate inputs about weight variation, the advantage of this system is that any extra goods or loads cannot be added while movement of the trucks, if any extra goods are been added then it can be alerted nonstop to the government agencies on the real-time or the weight on the trucks can be monitored by the central government organization. This paper will provide an integration of smart sensors and the Internet of Things (IoT) and Block Chain integration together on one platform, we propose in this paper using the block chain technology to encrypt the real-time data to the government database and this data is securely transferred or encapsulated in the block chain as block, these data cannot be modified by the cyber attacker.

Sign in / Sign up

Export Citation Format

Share Document