scholarly journals Real-Time Data Processing

2021 ◽  
pp. 147-156
Author(s):  
Fabiana Fournier ◽  
Inna Skarbovsky
Keyword(s):  
Decision Making ◽  
Real Time ◽  
Processing System ◽  
Event Processing ◽  
Time Data ◽  
System Architectures ◽  
Real Time Processing ◽  
Operational Decision Making ◽  
Real Time Data Processing ◽  
High Level

AbstractTo remain competitive, organizations are increasingly taking advantage of the high volumes of data produced in real time for actionable insights and operational decision-making. In this chapter, we present basic concepts in real-time analytics, their importance in today’s organizations, and their applicability to the bioeconomy domains investigated in the DataBio project. We begin by introducing key terminology for event processing, and motivation for the growing use of event processing systems, followed by a market analysis synopsis. Thereafter, we provide a high-level overview of event processing system architectures, with its main characteristics and components, followed by a survey of some of the most prominent commercial and open source tools. We then describe how we applied this technology in two of the DataBio project domains: agriculture and fishery. The devised generic pipeline for IoT data real-time processing and decision-making was successfully applied to three pilots in the project from the agriculture and fishery domains. This event processing pipeline can be generalized to any use case in which data is collected from IoT sensors and analyzed in real-time to provide real-time alerts for operational decision-making.

scholarly journals Bifrost: A Python/C++ Framework for High-Throughput Stream Processing in Astronomy

2017 ◽  
Vol 06 (04) ◽  
pp. 1750007 ◽  
Author(s):  
Miles D. Cranmer ◽  
Benjamin R. Barsdell ◽  
Danny C. Price ◽  
Jayce Dowell ◽  
Hugh Garsden ◽  
...  
Keyword(s):  
Real Time ◽  
High Throughput ◽  
Radio Astronomy ◽  
Scientific Productivity ◽  
Data Transport ◽  
Time Data ◽  
Process Data ◽  
Real Time Processing ◽  
Real Time Data Processing ◽  
High Level

Radio astronomy observatories with high throughput back end instruments require real-time data processing. While computing hardware continues to advance rapidly, development of real-time processing pipelines remains difficult and time-consuming, which can limit scientific productivity. Motivated by this, we have developed Bifrost: an open-source software framework for rapid pipeline development. (a) Bifrost combines a high-level Python interface with highly efficient reconfigurable data transport and a library of computing blocks for CPU and GPU processing. The framework is generalizable, but initially it emphasizes the needs of high-throughput radio astronomy pipelines, such as the ability to process data buffers as if they were continuous streams, the capacity to partition processing into distinct data sequences (e.g. separate observations), and the ability to extract specific intervals from buffered data. Computing blocks in the library are designed for applications such as interferometry, pulsar dedispersion and timing, and transient search pipelines. We describe the design and implementation of the Bifrost framework and demonstrate its use as the backbone in the correlation and beamforming back end of the Long Wavelength Array (LWA) station in the Sevilleta National Wildlife Refuge, NM.

FPGA+USB-Based Real-Time Data Processing System for Defects Inspection by CCD Scanning

2013 ◽  
Vol 278-280 ◽  
pp. 749-752 ◽  
Author(s):  
Peng Wang ◽  
Chi Zhong Wang ◽  
Ze Sen Liu ◽  
Xu Han ◽  
Cao Wang Si ◽  
...  
Keyword(s):  
Real Time ◽  
High Speed ◽  
Processing System ◽  
Data Processing System ◽  
Time Data ◽  
Defects Inspection ◽  
Signal Characteristics ◽  
Real Time Data ◽  
Real Time Data Processing ◽  
Hardware Circuit

In this paper, the real-time defects inspection was implemented via use of paralleled structure and high-speed operation of FPGA. The hardware circuit based on FPGA was established. According to signal characteristics of polymeric film defects, the preprocessing scheme of defect images based on FPGA was designed. The defect data were packed according to the defined format. Data processed were transferred to PC through USB2.0 real-timely to reconstruct defect microscopic images. The quantity of transferred data was decreased tremendously by this method. The inspecting speed was greatly improved.

scholarly journals A Framework for Real Time Processing of Sensor Data in the Cloud

2015 ◽  
Vol 2015 ◽  
pp. 1-11 ◽  
Author(s):  
Supun Kamburugamuve ◽  
Leif Christiansen ◽  
Geoffrey Fox
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Cloud Services ◽  
Sensor Data ◽  
Smart Devices ◽  
Time Data ◽  
Real Time Processing ◽  
Real Time Data ◽  
Autonomous Robot Navigation ◽  
Real Time Data Processing

We describe IoTCloud, a platform to connect smart devices to cloud services for real time data processing and control. A device connected to IoTCloud can communicate with real time data analysis frameworks deployed in the cloud via messaging. The platform design is scalable in connecting devices as well as transferring and processing data. With IoTCloud, a user can develop real time data processing algorithms in an abstract framework without concern for the underlying details of how the data is distributed and transferred. For this platform, we primarily consider real time robotics applications such as autonomous robot navigation, where there are strict requirements on processing latency and demand for scalable processing. To demonstrate the effectiveness of the system, a robotic application is developed on top of the framework. The system and the robotics application characteristics are measured to show that data processing in central servers is feasible for real time sensor applications.

scholarly journals Development of an On-line Real Time Data Processing System for the Ion Microanalyzer

1975 ◽  
Vol 23 (4) ◽  
pp. 271-279
Author(s):  
YOSHIAKI KATO ◽  
YOSHIO TAJIMA ◽  
HAJIME HAYAKAWA ◽  
ATSUSHI SHIBATA ◽  
KOUJI NISHIWAKI
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Processing System ◽  
Data Processing System ◽  
Time Data ◽  
Real Time Data ◽  
On Line ◽  
Real Time Data Processing
Sign in / Sign up

Export Citation Format

Share Document