scholarly journals Bifrost: A Python/C++ Framework for High-Throughput Stream Processing in Astronomy

2017 ◽  
Vol 06 (04) ◽  
pp. 1750007 ◽  
Author(s):  
Miles D. Cranmer ◽  
Benjamin R. Barsdell ◽  
Danny C. Price ◽  
Jayce Dowell ◽  
Hugh Garsden ◽  
...  
Keyword(s):  
Real Time ◽  
High Throughput ◽  
Radio Astronomy ◽  
Scientific Productivity ◽  
Data Transport ◽  
Time Data ◽  
Process Data ◽  
Real Time Processing ◽  
Real Time Data Processing ◽  
High Level

Radio astronomy observatories with high throughput back end instruments require real-time data processing. While computing hardware continues to advance rapidly, development of real-time processing pipelines remains difficult and time-consuming, which can limit scientific productivity. Motivated by this, we have developed Bifrost: an open-source software framework for rapid pipeline development. (a) Bifrost combines a high-level Python interface with highly efficient reconfigurable data transport and a library of computing blocks for CPU and GPU processing. The framework is generalizable, but initially it emphasizes the needs of high-throughput radio astronomy pipelines, such as the ability to process data buffers as if they were continuous streams, the capacity to partition processing into distinct data sequences (e.g. separate observations), and the ability to extract specific intervals from buffered data. Computing blocks in the library are designed for applications such as interferometry, pulsar dedispersion and timing, and transient search pipelines. We describe the design and implementation of the Bifrost framework and demonstrate its use as the backbone in the correlation and beamforming back end of the Long Wavelength Array (LWA) station in the Sevilleta National Wildlife Refuge, NM.

scholarly journals Real-Time Data Processing

2021 ◽  
pp. 147-156
Author(s):  
Fabiana Fournier ◽  
Inna Skarbovsky
Keyword(s):  
Decision Making ◽  
Real Time ◽  
Processing System ◽  
Event Processing ◽  
Time Data ◽  
System Architectures ◽  
Real Time Processing ◽  
Operational Decision Making ◽  
Real Time Data Processing ◽  
High Level

AbstractTo remain competitive, organizations are increasingly taking advantage of the high volumes of data produced in real time for actionable insights and operational decision-making. In this chapter, we present basic concepts in real-time analytics, their importance in today’s organizations, and their applicability to the bioeconomy domains investigated in the DataBio project. We begin by introducing key terminology for event processing, and motivation for the growing use of event processing systems, followed by a market analysis synopsis. Thereafter, we provide a high-level overview of event processing system architectures, with its main characteristics and components, followed by a survey of some of the most prominent commercial and open source tools. We then describe how we applied this technology in two of the DataBio project domains: agriculture and fishery. The devised generic pipeline for IoT data real-time processing and decision-making was successfully applied to three pilots in the project from the agriculture and fishery domains. This event processing pipeline can be generalized to any use case in which data is collected from IoT sensors and analyzed in real-time to provide real-time alerts for operational decision-making.

scholarly journals A Framework for Real Time Processing of Sensor Data in the Cloud

2015 ◽  
Vol 2015 ◽  
pp. 1-11 ◽  
Author(s):  
Supun Kamburugamuve ◽  
Leif Christiansen ◽  
Geoffrey Fox
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Cloud Services ◽  
Sensor Data ◽  
Smart Devices ◽  
Time Data ◽  
Real Time Processing ◽  
Real Time Data ◽  
Autonomous Robot Navigation ◽  
Real Time Data Processing

We describe IoTCloud, a platform to connect smart devices to cloud services for real time data processing and control. A device connected to IoTCloud can communicate with real time data analysis frameworks deployed in the cloud via messaging. The platform design is scalable in connecting devices as well as transferring and processing data. With IoTCloud, a user can develop real time data processing algorithms in an abstract framework without concern for the underlying details of how the data is distributed and transferred. For this platform, we primarily consider real time robotics applications such as autonomous robot navigation, where there are strict requirements on processing latency and demand for scalable processing. To demonstrate the effectiveness of the system, a robotic application is developed on top of the framework. The system and the robotics application characteristics are measured to show that data processing in central servers is feasible for real time sensor applications.

scholarly journals DESIGN AND IMPLEMENTATION OF A LOW-COST UAV-BASED MULTI-SENSOR PAYLOAD FOR RAPID-RESPONSE MAPPING APPLICATIONS

2016 ◽  
Vol XLI-B1 ◽  
pp. 1017-1022 ◽  
Author(s):  
M. Sakr ◽  
Z. Lari ◽  
N. El-Sheimy
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Low Cost ◽  
Building Blocks ◽  
Rapid Response ◽  
Disaster Mitigation ◽  
Time Data ◽  
Real Time Processing ◽  
Real Time Data ◽  
Real Time Data Processing

The main objective of this paper is to investigate the potential of using Unmanned Aerial Vehicles (UAVs) as a platform to collect geospatial data for rapid response applications, especially in hard-to-access and hazardous areas. The UAVs are low-cost mapping vehicles, and they are easy to handle and deploy in-field. These characteristics make UAVs ideal candidates for rapid-response and disaster mitigation scenarios. The majority of the available UAV systems are not capable of real-time/near real-time data processing. This paper introduces a low-cost UAV-based multi-sensor mapping payload which supports real-time processing and can be effectively used in rapid-response applications. The paper introduces the main components of the system, and provides an overview of the proposed payload architecture. Then, it introduces the implementation details of the major building blocks of the system. Finally, the paper presents our conclusions and the future work, in order to achieve real-time/near real-time data processing and product delivery capabilities.

scholarly journals DESIGN AND IMPLEMENTATION OF A LOW-COST UAV-BASED MULTI-SENSOR PAYLOAD FOR RAPID-RESPONSE MAPPING APPLICATIONS

2016 ◽  
Vol XLI-B1 ◽  
pp. 1017-1022
Author(s):  
M. Sakr ◽  
Z. Lari ◽  
N. El-Sheimy
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Low Cost ◽  
Building Blocks ◽  
Rapid Response ◽  
Disaster Mitigation ◽  
Time Data ◽  
Real Time Processing ◽  
Real Time Data ◽  
Real Time Data Processing

The main objective of this paper is to investigate the potential of using Unmanned Aerial Vehicles (UAVs) as a platform to collect geospatial data for rapid response applications, especially in hard-to-access and hazardous areas. The UAVs are low-cost mapping vehicles, and they are easy to handle and deploy in-field. These characteristics make UAVs ideal candidates for rapid-response and disaster mitigation scenarios. The majority of the available UAV systems are not capable of real-time/near real-time data processing. This paper introduces a low-cost UAV-based multi-sensor mapping payload which supports real-time processing and can be effectively used in rapid-response applications. The paper introduces the main components of the system, and provides an overview of the proposed payload architecture. Then, it introduces the implementation details of the major building blocks of the system. Finally, the paper presents our conclusions and the future work, in order to achieve real-time/near real-time data processing and product delivery capabilities.

scholarly journals Real-Time High-Level Acute Pain Detection Using a Smartphone and a Wrist-Worn Electrodermal Activity Sensor

Sensors ◽  
2021 ◽  
Vol 21 (12) ◽  
pp. 3956
Author(s):  
Youngsun Kong ◽  
Hugo F. Posada-Quintero ◽  
Ki H. Chon
Keyword(s):  
Real Time ◽  
Electrodermal Activity ◽  
Smartphone Application ◽  
Real Time Processing ◽  
Pain Stimulation ◽  
Significant Difference ◽  
Before And After ◽  
Electrical Pulses ◽  
Pain Detection ◽  
High Level

The subjectiveness of pain can lead to inaccurate prescribing of pain medication, which can exacerbate drug addiction and overdose. Given that pain is often experienced in patients’ homes, there is an urgent need for ambulatory devices that can quantify pain in real-time. We implemented three time- and frequency-domain electrodermal activity (EDA) indices in our smartphone application that collects EDA signals using a wrist-worn device. We then evaluated our computational algorithms using thermal grill data from ten subjects. The thermal grill delivered a level of pain that was calibrated for each subject to be 8 out of 10 on a visual analog scale (VAS). Furthermore, we simulated the real-time processing of the smartphone application using a dataset pre-collected from another group of fifteen subjects who underwent pain stimulation using electrical pulses, which elicited a VAS pain score level 7 out of 10. All EDA features showed significant difference between painless and pain segments, termed for the 5-s segments before and after each pain stimulus. Random forest showed the highest accuracy in detecting pain, 81.5%, with 78.9% sensitivity and 84.2% specificity with leave-one-subject-out cross-validation approach. Our results show the potential of a smartphone application to provide near real-time objective pain detection.

Sign in / Sign up

Export Citation Format

Share Document