Comparative analysis of real-time messages in big data pipeline architecture

Author(s):  
Thandar Aung ◽  
Hla Yin Min ◽  
Aung Htein Maw
2021 ◽  
Author(s):  
Maximilian Georg Schuberth ◽  
Håkon Sunde Bakka ◽  
Claire Emma Birnie ◽  
Stefan Dümmong ◽  
Kjetil Eik Haavik ◽  
...  

Abstract Fiber Optic (FO) sensing capabilities for downhole monitoring include, among other techniques, Distributed Temperature Sensing (DTS) and Distributed Acoustic Sensing (DAS). The appeal of DTS and DAS data is based on its high temporal and spatial sampling, allowing for very fine localization of processes in a wellbore. Furthermore, the broad frequency spectrum that especially DAS data is acquired with, enables observations, ranging from more continuous effects like oil flow, to more distinct effects like opening and closing of valves. Due to the high data volume of hundreds of Gb per well per hour, DAS data has traditionally been acquired acquisition-based, where data is recorded for a limited amount of time and processed at a later point in time. This limits the decision-making capability based on this data as reacting to events is only possible long after the event occurred. Equinor has addressed these decision-making shortcomings by building a real-time streaming solution for transferring, processing, and interpretation of its FO data at the Johan Sverdrup field in the North Sea. The streaming solution for FO data consists of offshore interrogators streaming raw DAS and DTS data via a dedicated bandwidth to an onshore processing cluster. There, DAS data is transformed into FO feature data, e.g., Frequency Band Energies, which are heavily decimated versions of the raw data; allowing insight extraction, while significantly reducing data volumes. DTS and DAS FO feature data are then streamed to a custom-made, cloud-based visualization and integration platform. This cloud-based platform allows efficient inspection of large data sets, control and evaluation of applications based on these data, and sharing of FO data within the Johan Sverdrup asset. During the last year, this FO data streaming pipeline has processed several tens of PB of FO data, monitoring a range of well operations and processes. Qualitatively, the benefits and potential of the real-time data acquisitions have been illustrated by providing a greater understanding of current well conditions and processes. Alongside the FO data pipeline, multiple prototype applications have been developed for automated monitoring of Gas Lift Valves, Safety Valve operations, Gas Lift rate estimation, and monitoring production start-up, all providing insights in real-time. For certain use cases, such as monitoring production start-up, the FO data provides a previously non-existent monitoring solution. In this paper, we will discuss in detail the FO data pipeline architecture from-platform-to-cloud, illustrate several data examples, and discuss the way-forward for "real-time" FO data analytics.


2020 ◽  
Vol 3 (1) ◽  
Author(s):  
Angelina Pramana Thenata

The rapid development of information has made online news increasingly needed. Online news attracts readers' attention by providing convenience and speed in presenting news from various fields. However, the large amount (volume) of online news that spreads in a short time (velocity) and the public's need to consume news in various references (variety) can affect people's lives. Therefore, the government as the regulator and news agencies need to monitor online news circulating. Based on these problems, the researcher proposes a data lake architectural design that is suitable for online news and can run in real-time. Data lakes can solve the main problems of Big Data (volume, velocity, variety). In proposing this data lake architecture, the researcher conducted a literature study and analyzed the flow of the data lake architecture according to online news. Furthermore, the researcher will use this architecture to combine and uniform the online news data structure from several online news channels and then stream it in real-time to fill the data lake. The results of using the data lake architecture for online news will be stored on MongoDB which functions as a database to store all data for both the short and long term. Finally, this data lake will be a means to accommodate, dive into, and analyze the circulating online news data. Keywords – Data Lake, Online News, Real-Time.


Author(s):  
V. Нolovan ◽  
V. Gerasimov ◽  
А. Нolovan ◽  
N. Maslich

Fighting in the Donbas, which has been going on for more than five years, shows that a skillful counter-battery fight is an important factor in achieving success in wars of this kind. Especially in conditions where for the known reasons the use of combat aviation is minimized. With the development of technical warfare, the task of servicing the counter-battery fight began to rely on radar stations (radar) to reconnaissance the positions of artillery, which in modern terms are called counter-battery radar. The principle of counter-battery radar is based on the detection of a target (artillery shell, mortar mine or rocket) in flight at an earlier stage and making several measurements of the coordinates of the current position of the ammunition. According to these data, the trajectory of the projectile's flight is calculated and, on the basis of its prolongation and extrapolation of measurements, the probable coordinates of the artillery, as well as the places of ammunition falling, are determined. In addition, the technical capabilities of radars of this class allow you to recognize the types and caliber of artillery systems, as well as to adjust the fire of your artillery. The main advantages of these radars are:  mobility (transportability);  inspection of large tracts of terrain over long distances;  the ability to obtain target's data in near real-time;  independence from time of day and weather conditions;  relatively high fighting efficiency. The purpose of the article is to determine the leading role and place of the counter-battery radar among other artillery instrumental reconnaissance tools, to compare the combat capabilities of modern counter-battery radars, armed with Ukrainian troops and some leading countries (USA, China, Russia), and are being developed and tested in Ukraine. The method of achieving this goal is a comparative analysis of the features of construction and combat capabilities of modern models of counter-battery radar in Ukraine and in other countries. As a result of the conducted analysis, the directions of further improvement of the radar armament, increasing the capabilities of existing and promising counter-battery radar samples were determined.


Healthcare ◽  
2020 ◽  
Vol 8 (3) ◽  
pp. 234 ◽  
Author(s):  
Hyun Yoo ◽  
Soyoung Han ◽  
Kyungyong Chung

Recently, a massive amount of big data of bioinformation is collected by sensor-based IoT devices. The collected data are also classified into different types of health big data in various techniques. A personalized analysis technique is a basis for judging the risk factors of personal cardiovascular disorders in real-time. The objective of this paper is to provide the model for the personalized heart condition classification in combination with the fast and effective preprocessing technique and deep neural network in order to process the real-time accumulated biosensor input data. The model can be useful to learn input data and develop an approximation function, and it can help users recognize risk situations. For the analysis of the pulse frequency, a fast Fourier transform is applied in preprocessing work. With the use of the frequency-by-frequency ratio data of the extracted power spectrum, data reduction is performed. To analyze the meanings of preprocessed data, a neural network algorithm is applied. In particular, a deep neural network is used to analyze and evaluate linear data. A deep neural network can make multiple layers and can establish an operation model of nodes with the use of gradient descent. The completed model was trained by classifying the ECG signals collected in advance into normal, control, and noise groups. Thereafter, the ECG signal input in real time through the trained deep neural network system was classified into normal, control, and noise. To evaluate the performance of the proposed model, this study utilized a ratio of data operation cost reduction and F-measure. As a result, with the use of fast Fourier transform and cumulative frequency percentage, the size of ECG reduced to 1:32. According to the analysis on the F-measure of the deep neural network, the model had 83.83% accuracy. Given the results, the modified deep neural network technique can reduce the size of big data in terms of computing work, and it is an effective system to reduce operation time.


Sign in / Sign up

Export Citation Format

Share Document