scholarly journals A scientific data processing framework for time series NetCDF data

2014 ◽  
Vol 60 ◽  
pp. 241-249 ◽  
Author(s):  
Krista Gaustad ◽  
Tim Shippert ◽  
Brian Ermold ◽  
Sherman Beus ◽  
Jeff Daily ◽  
...  
Keyword(s):  
Time Series ◽  
Data Processing ◽  
Scientific Data ◽  
Processing Framework

Big traffic data processing framework for intelligent monitoring and recording systems

2016 ◽  
Vol 181 ◽  
pp. 139-146 ◽  
Author(s):  
Yingjie Xia ◽  
Jinlong Chen ◽  
Xindai Lu ◽  
Chunhui Wang ◽  
Chao Xu
Keyword(s):  
Data Processing ◽  
Traffic Data ◽  
Intelligent Monitoring ◽  
Big Traffic Data ◽  
Processing Framework

scholarly journals Real Time Data Processing Framework

2015 ◽  
Vol 5 (5) ◽  
pp. 49-63 ◽  
Author(s):  
Karan Patel ◽  
Yash Sakaria ◽  
Chetashri Bhadane
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Time Data ◽  
Real Time Data ◽  
Real Time Data Processing ◽  
Processing Framework

scholarly journals Highly comparative time-series analysis: the empirical structure of time series and their methods

2013 ◽  
Vol 10 (83) ◽  
pp. 20130048 ◽  
Author(s):  
Ben D. Fulcher ◽  
Max A. Little ◽  
Nick S. Jones
Keyword(s):  
Time Series ◽  
Time Series Analysis ◽  
Time Series Data ◽  
Scientific Data ◽  
Series Data ◽  
Series Analysis ◽  
Analysis Methods ◽  
Scientific Disciplines ◽  
History Of ◽  
Reduced Representations

The process of collecting and organizing sets of observations represents a common theme throughout the history of science. However, despite the ubiquity of scientists measuring, recording and analysing the dynamics of different processes, an extensive organization of scientific time-series data and analysis methods has never been performed. Addressing this, annotated collections of over 35 000 real-world and model-generated time series, and over 9000 time-series analysis algorithms are analysed in this work. We introduce reduced representations of both time series, in terms of their properties measured by diverse scientific methods, and of time-series analysis methods, in terms of their behaviour on empirical time series, and use them to organize these interdisciplinary resources. This new approach to comparing across diverse scientific data and methods allows us to organize time-series datasets automatically according to their properties, retrieve alternatives to particular analysis methods developed in other scientific disciplines and automate the selection of useful methods for time-series classification and regression tasks. The broad scientific utility of these tools is demonstrated on datasets of electroencephalograms, self-affine time series, heartbeat intervals, speech signals and others, in each case contributing novel analysis techniques to the existing literature. Highly comparative techniques that compare across an interdisciplinary literature can thus be used to guide more focused research in time-series analysis for applications across the scientific disciplines.

scholarly journals THE GWAC DATA PROCESSING AND MANAGEMENT SYSTEM

2021 ◽  
Vol 53 ◽  
pp. 174-179
Author(s):  
Y. Xu ◽  
L. P. Xin ◽  
X. H. Han ◽  
H. B. Cai ◽  
L. Huang ◽  
...  
Keyword(s):  
Data Processing ◽  
User Interfaces ◽  
Management System ◽  
Scientific Data ◽  
Data Archiving ◽  
Processing Pipeline ◽  
Advanced Technologies ◽  
Single Frame ◽  
Moonless Night ◽  
Processing Requirement

GWAC will have been built an integrated FOV of 5,000 degree2 and have already built 1,800 square degree2. The limit magnitude of a 10-second exposure image in the moonless night is 16R. In each observation night, GWAC produces about 0.7TB of raw data, and the data processing pipeline generates millions of single frame alerts. We describe the GWAC Data Processing and Management System (GPMS), including hardware architecture, database, detection-filtering-validation of transient candidates, data archiving, and user interfaces for the check of transient and the monitor of the system. GPMS combines general technology and software in astronomy and computer field, and use some advanced technologies such as deep learning. Practical results show that GPMS can fully meet the scientific data processing requirement of GWAC. It can online accomplish the detection, filtering and validation of millions of transient candidates, and feedback the final results to the astronomer in real-time. During the observation from October of 2018 to December of 2019, we have already found 102 transients.

Regression derivatives and their application for registration of a tsunami wave arrival by sea-level records

2019 ◽  
pp. 28-41
Author(s):  
Сергей Мартикович Агаян ◽  
Шамиль Рафекович Богоутдинов ◽  
Ольга Васильевна Иванченко ◽  
Дмитрий Альфредович Камаев
Keyword(s):  
Time Series ◽  
Data Processing ◽  
Sea Level ◽  
Discrete Time ◽  
Mathematical Analysis ◽  
Visual Analysis ◽  
Tsunami Wave ◽  
Information Support ◽  
Discrete Time Series ◽  
Wave Arrival

Структура дискретного временного ряда тесно связана со свойствами процесса, который он описывает. В рамках дискретного математического анализа имеется несколько подходов к анализу структуры дискретных рядов: геометрические меры, динамические коридоры и концепция тренда. Для дискретного временного ряда, заданного в общем случае на нерегулярной сетке, с характером тренда тесным образом связана регрессионная производная: области ее положительного (отрицательного) значения соответствуют возрастающим (убывающим) трендам, а границы между ними - экстремумам. В настоящей работе исследуются возможности применения методов дискретного математического анализа для разработки процедуры регистрации вступления волны цунами по оперативным данным измерения уровня моря. The research addresses the possibility of application of the methods of discrete mathematical analysis to develop a procedure for recording tsunami wave arrival on the base of the operational data for measuring sea level. As a basis for constructing a tsunami wave registration procedure, this research uses a schematization of the actions of the oceanographer on-duty during visual analysis of the sea level records. The task of automatic registration of a tsunami wave by sea level recording arises in various situations of information support of the oceanographer on duty. Requirements for the processing of sea level records depend on the situation. The structure of a discrete time series is closely related to the properties of the described process. As part of the discrete mathematical analysis, there are several approaches to the analysis of the structure of discrete series: geometric measures, dynamic corridors and the trend concept. For a discrete time series, given in the general case on an irregular grid, the regression derivative is closely related to the nature of the trend: the areas of its positive (negative) values correspond to the increasing (decreasing) trends, and the boundaries between them are extremes. The content of this research is a presentation of data processing techniques using regression derivatives, constructing data processing procedures based on derivatives, as well as a demonstration of their applicability to the problem of recording tsunami wave arrival according to the measuring of sea level.

Theoretical Consideration on the Application of Continuous Wavelet Transform to Time-Series Processing for Magnetotelluric Data

2020 ◽  
Author(s):  
Hiroki Ogawa ◽  
Yuki Hama ◽  
Koichi Asamori ◽  
Takumi Ueda
Keyword(s):  
Time Series ◽  
Fourier Transform ◽  
Data Processing ◽  
Time Series Data ◽  
Series Data ◽  
Short Time Fourier Transform ◽  
Numerical Errors ◽  
Spectral Transform ◽  
Frequency Components ◽  
Short Time

Abstract In the magnetotelluric (MT) method, the responses of the natural electromagnetic fields are evaluated by transforming time-series data into spectral data and calculating the apparent resistivity and phase. The continuous wavelet transform (CWT) can be an alternative to the short-time Fourier transform, and the applicability of CWT to MT data has been reported. There are, however, few cases of considering the effect of numerical errors derived from spectral transform on MT data processing. In general, it is desirable to adopt a window function narrow in the time domain for higher-frequency components and one in the frequency domain for lower-frequency components. In conducting the short-time Fourier transform, because the size of the window function is fixed unless the time-series data are decimated, there might be difference between the calculated MT responses and the true ones due to the numerical errors. Meanwhile, CWT can strike a balance between the resolution of the time and frequency domains by magnifying or reducing the wavelet, according to the value of frequency. Although the types of wavelet functions and their parameters influence the resolution of time and frequency, those calculation settings of CWT are often determined empirically. In this study, focusing on the frequency band between 0.001 Hz and 10 Hz, we demonstrated the superiority of utilizing CWT in MT data processing and determined its proper calculation settings in terms of restraining the numerical errors caused by the spectral transform of time-series data. The results obtained with the short-time Fourier transform accompanied with gradual decimation of the time-series data, called cascade decimation, were compared with those of CWT. The shape of the wavelet was changed by using different types of wavelet functions or their parameters, and the respective results of data processing were compared. Through these experiments, this study indicates that CWT with the complex Morlet function with its wavelet parameter k set to 6 ≤ k < 10 will be effective in restraining the numerical errors caused by the spectral transform.

Sign in / Sign up

Export Citation Format

Share Document