Bandwidth compression of multiple numerical data streams for high performance custom computing

2014 ◽  
Author(s):  
Tomohiro Ueno ◽  
Ryo Ito ◽  
Kentaro Sano ◽  
Satoru Yamamoto
Keyword(s):  
Data Streams ◽  
High Performance ◽  
Numerical Data ◽  
Bandwidth Compression

Using computer clusters for processing spatial-temporal data streams in data acquisition systems

2018 ◽  
Vol 935 (5) ◽  
pp. 54-63
Author(s):  
A.A. Maiorov ◽  
A.V. Materuhin ◽  
I.N. Kondaurov
Keyword(s):  
Data Acquisition ◽  
Data Streams ◽  
Spatial Data ◽  
High Performance ◽  
Temporal Data ◽  
Cyberphysical Systems ◽  
Software Environment ◽  
Computer Clusters ◽  
Data Acquisition Systems ◽  
Essential Components

Geoinformation technologies are now becoming “end-to-end” technologies of the new digital economy. There is a need for solutions for efficient processing of spatial and spatio-temporal data that could be applied in various sectors of this new economy. Such solutions are necessary, for example, for cyberphysical systems. Essential components of cyberphysical systems are high-performance and easy-scalable data acquisition systems based on smart geosensor networks. This article discusses the problem of choosing a software environment for this kind of systems, provides a review and a comparative analysis of various open source software environments designed for large spatial data and spatial-temporal data streams processing in computer clusters. It is shown that the software framework STARK can be used to process spatial-temporal data streams in spatial-temporal data streams. An extension of the STARK class system based on the type system for spatial-temporal data streams developed by one of the authors of this article is proposed. The models and data representations obtained as a result of the proposed expansion can be used not only for processing spatial-temporal data streams in data acquisition systems based on smart geosensor networks, but also for processing spatial-temporal data streams in various purposes geoinformation systems that use processing data in computer clusters.

scholarly journals Bonsai: An event-based framework for processing and controlling data streams

2014 ◽  
Author(s):  
Gonçalo Lopes ◽  
Niccolò Bonacchi ◽  
João Frazão ◽  
Joana P. Neto ◽  
Bassam V. Atallah ◽  
...  
Keyword(s):  
Data Streams ◽  
High Performance ◽  
Visual Programming ◽  
Scientific Data ◽  
Video Tracking ◽  
Loop Control ◽  
Online Processing ◽  
Parallel Data ◽  
Develop Software ◽  
Event Based

The design of modern scientific experiments requires the control and monitoring of many parallel data streams. However, the serial execution of programming instructions in a computer makes it a challenge to develop software that can deal with the asynchronous, parallel nature of scientific data. Here we present Bonsai, a modular, high-performance, open-source visual programming framework for the acquisition and online processing of data streams. We describe Bonsai's core principles and architecture and demonstrate how it allows for flexible and rapid prototyping of integrated experimental designs in neuroscience. We specifically highlight different possible applications which require the combination of many different hardware and software components, including behaviour video tracking, electrophysiology and closed-loop control of stimulation parameters.

A Tour of Lattice-Based Skyline Algorithms

2018 ◽  
pp. 96-122
Author(s):  
Markus Endres ◽  
Lena Rudenko
Keyword(s):  
Real Time ◽  
Data Streams ◽  
High Performance ◽  
State Of The Art ◽  
Experimental Results ◽  
Lattice Structures ◽  
Skyline Query ◽  
Basic Concepts ◽  
Generic Index

A skyline query retrieves all objects in a dataset that are not dominated by other objects according to some given criteria. There exist many skyline algorithms which can be classified into generic, index-based, and lattice-based algorithms. This chapter takes a tour through lattice-based skyline algorithms. It summarizes the basic concepts and properties, presents high-performance parallel approaches, shows how one overcomes the low-cardinality restriction of lattice structures, and finally presents an application on data streams for real-time skyline computation. Experimental results on synthetic and real datasets show that lattice-based algorithms outperform state-of-the-art skyline techniques, and additionally have a linear runtime complexity.

Dynamic Investigation on a Pelton Runner: FEM Calculation and Experimental Results

2007 ◽  
Author(s):  
Emanuel Pesatori ◽  
Giorgio Turozzi ◽  
Paolo Pennacchi ◽  
Alessandro Tosi
Keyword(s):  
High Performance ◽  
Natural Frequencies ◽  
Numerical Data ◽  
Point Of View ◽  
Normal Operation ◽  
Mode Shapes ◽  
Modal Behavior ◽  
Dynamic Investigation ◽  
Strong Dynamic ◽  
Structural Point

The dynamical phenomena involving a Pelton runner during normal operation are getting more and more of concern in the design of such kind of turbines. In fact, the increasing of the performance required by the market imposes runner designs that can become critical from a structural point of view. Each bucket is subjected to a certain number of impulses per revolution corresponding to the number of nozzles. This high frequency load could lead to a strong dynamic response of the runner if some kind of resonance takes place. For these reasons, an accurate modal analysis is to be performed before the runner realization, in order to avoid any crack arising during operation. This paper describes how a new, high performance profiled Pelton runner was modeled and dynamically calculated by FEM code ANSYS in order to obtain its natural frequencies and mode shapes. Subsequently, the modal behavior of the runner was experimentally investigated by means of an impulse test on field. The natural frequencies were got by examinating the FFT of the response and the mode shapes were reconstructed according to the stored signals of the accelerometers. The match between the experimental and numerical data was pretty good.

scholarly journals shinyheatmap: ultra fast low memory heatmap web interface for big data genomics

2016 ◽  
Author(s):  
Bohdan B. Khomtchouk ◽  
James R. Hennessy ◽  
Claes Wahlestedt
Keyword(s):  
Big Data ◽  
High Performance ◽  
Numerical Data ◽  
Large Datasets ◽  
Barriers To Entry ◽  
Web Interface ◽  
Link Type ◽  
Sequencing Studies ◽  
Performance Benchmarks ◽  
Computational Resources

AbstractBackgroundTranscriptomics, metabolomics, metagenomics, and other various next-generation sequencing (-omics) fields are known for their production of large datasets, especially across single-cell sequencing studies. Visualizing such big data has posed technical challenges in biology, both in terms of available computational resources as well as programming acumen. Since heatmaps are used to depict high-dimensional numerical data as a colored grid of cells, efficiency and speed have often proven to be critical considerations in the process of successfully converting data into graphics. For example, rendering interactive heatmaps from large input datasets (e.g., 100k+ rows) has been computationally infeasible on both desktop computers and web browsers. In addition to memory requirements, programming skills and knowledge have frequently been barriers-to-entry for creating highly customizable heatmaps.ResultsWe propose shinyheatmap: an advanced user-friendly heatmap software suite capable of efficiently creating highly customizable static and interactive biological heatmaps in a web browser. shinyheatmap is a low memory footprint program, making it particularly well-suited for the interactive visualization of extremely large datasets that cannot typically be computed in-memory due to size restrictions. Also, shinyheatmap features a built-in high performance web plug-in, fastheatmap, for rapidly plotting interactive heatmaps of datasets as large as 105 − 107 rows within seconds, effectively shattering previous performance benchmarks of heatmap rendering speed.Conclusionsshinyheatmap is hosted online as a freely available web server with an intuitive graphical user interface: http://shinyheatmap.com. The methods are implemented in R, and are available as part of the shinyheatmap project at: https://github.com/Bohdan-Khomtchouk/shinyheatmap. Users can access fastheatmap directly from within the shinyheatmap web interface, and all source code has been made publicly available on Github: https://github.com/Bohdan-Khomtchouk/fastheatmap.

In the land of data streams where synopses are missing, one framework to bring them all

2021 ◽  
Vol 14 (10) ◽  
pp. 1818-1831
Author(s):  
Rudi Poepsel-Lemaitre ◽  
Martin Kiefer ◽  
Joscha von Hein ◽  
Jorge-Arnulfo Quiané-Ruiz ◽  
Volker Markl
Keyword(s):  
Data Analysis ◽  
Real Time ◽  
Data Streams ◽  
High Performance ◽  
Stream Processing ◽  
Divide And Conquer ◽  
Time Data ◽  
Real Time Data ◽  
Internal Processing ◽  
Aggregate Functions

In pursuit of real-time data analysis, approximate summarization structures, i.e., synopses, have gained importance over the years. However, existing stream processing systems, such as Flink, Spark, and Storm, do not support synopses as first class citizens, i.e., as pipeline operators. Synopses' implementation is upon users. This is mainly because of the diversity of synopses, which makes a unified implementation difficult. We present Condor, a framework that supports synopses as first class citizens. Condor facilitates the specification and processing of synopsis-based streaming jobs while hiding all internal processing details. Condor's key component is its model that represents synopses as a particular case of windowed aggregate functions. An inherent divide and conquer strategy allows Condor to efficiently distribute the computation, allowing for high-performance and linear scalability. Our evaluation shows that Condor outperforms existing approaches by up to a factor of 75x and that it scales linearly with the number of cores.

Merging Multiple Data Streams on Common Keys over High Performance Networks

2002 ◽  
Author(s):  
M. Mazzucco ◽  
A. Ananthanarayan ◽  
R.L. Grossman ◽  
J. Levera ◽  
G. Bhagavantha Rao
Keyword(s):  
Data Streams ◽  
High Performance ◽  
Multiple Data ◽  
Multiple Data Streams
Sign in / Sign up

Export Citation Format

Share Document