Implementing a new data acquisition system for the advanced integrated atmospheric observation system KITcube

2020 ◽  
Author(s):  
Martin Kohler ◽  
Mahnaz Fekri ◽  
Andreas Wieser ◽  
Jan Handwerker
Keyword(s):  
Time Series ◽  
Data Acquisition ◽  
Real Time ◽  
Binary Data ◽  
Data Exchange ◽  
Data Access ◽  
Series Data ◽  
Observation System ◽  
Observation Data ◽  
File Server

<p>KITcube (Kalthoff et al, 2013) is a mobile advanced integrated observation system for the measurement of meteorological processes within a volume of 10x10x10 km<sup>3</sup>. A large variety of different instruments from in-situ sensors to scanning remote sensing devices are deployed during campaigns. The simultaneous operation and real time instrument control needed for maximum instrument synergy requires a real-time data management designed to cover the various user needs: Save data acquisition, fast loading, compressed storage, easy data access, monitoring and data exchange. Large volumes of data such as raw and semi-processed data of various data types, from simple ASCII time series to high frequency multi-dimensional binary data provide abundant information, but makes the integration and efficient management of such data volumes to a challenge.<br>Our data processing architecture is based on open source technologies and involves the following five sections: 1) Transferring: Data and metadata collected during a campaign are stored on a file server. 2) Populating the database: A relational database is used for time series data and a hybrid database model for very large, complex, unstructured data. 3) Quality control: Automated checks for data acceptance and data consistency. 4) Monitoring: Data visualization in a web-application. 5) Data exchange: Allows the exchange of observation data and metadata in specified data formats with external users.<br>The implemented data architecture and workflow is illustrated in this presentation using data from the MOSES project (http://moses.eskp.de/home).</p><p>References:</p><p><strong>KITcube - A mobile observation platform for convection studies deployed during HyMeX </strong>.<br>Kalthoff, N.; Adler, B.; Wieser, A.; Kohler, M.; Träumner, K.; Handwerker, J.; Corsmeier, U.; Khodayar, S.; Lambert, D.; Kopmann, A.; Kunka, N.; Dick, G.; Ramatschi, M.; Wickert, J.; Kottmeier, C.<br>2013. Meteorologische Zeitschrift, 22 (6), 633–647. doi:10.1127/0941-2948/2013/0542 </p>

Accessing environmental time series data in R from Sensor Observation Services with ease

2020 ◽  
Author(s):  
Daniel Nüst ◽  
Eike H. Jürrens ◽  
Benedikt Gräler ◽  
Simon Jirka
Keyword(s):  
Time Series ◽  
Time Series Data ◽  
Data Access ◽  
R Package ◽  
Series Data ◽  
Observation Data ◽  
Software Environment ◽  
Technical Standards ◽  
Seamless Integration ◽  
Environmental Time Series

<p>Time series data of in-situ measurements is the key to many environmental studies. The first challenge in any analysis typically arises when the data needs to be imported into the analysis framework. Standardisation is one way to lower this burden. Unfortunately, relevant interoperability standards might be challenging for non-IT experts as long as they are not dealt with behind the scenes of a client application. One standard to provide access to environmental time series data is the Sensor Observation Service (SOS, ) specification published by the Open Geospatial Consortium (OGC). SOS instances are currently used in a broad range of applications such as hydrology, air quality monitoring, and ocean sciences. Data sets provided via an SOS interface can be found around the globe from Europe to New Zealand.</p><p>The R package sos4R (Nüst et al., 2011) is an extension package for the R environment for statistical computing and visualization (), which has been demonstrated a a powerful tools for conducting and communicating geospatial research (cf. Pebesma et al., 2012; ). sos4R comprises a client that can connect to an SOS server. The user can use it to query data from SOS instances using simple R function calls. It provides a convenience layer for R users to integrate observation data from data access servers compliant with the SOS standard without any knowledge about the underlying technical standards. To further improve the usability for non-SOS experts, a recent update to sos4R includes a set of wrapper functions, which remove complexity and technical language specific to OGC specifications. This update also features specific consideration of the OGC SOS 2.0 Hydrology Profile and thereby opens up a new scientific domain.</p><p>In our presentation we illustrate use cases and examples building upon sos4R easing the access of time series data in an R and Shiny () context. We demonstrate how the abstraction provided in the client library makes sensor observation data for accessible and further show how sos4R allows the seamless integration of distributed observations data, i.e., across organisational boundaries, into transparent and reproducible data analysis workflows.</p><p><strong>References</strong></p><p>Nüst D., Stasch C., Pebesma E. (2011) Connecting R to the Sensor Web. In: Geertman S., Reinhardt W., Toppen F. (eds) Advancing Geoinformation Science for a Changing World. Lecture Notes in Geoinformation and Cartography, Springer. </p><p>Pebesma, E., Nüst, D., & Bivand, R. (2012). The R software environment in reproducible geoscientific research. Eos, Transactions American Geophysical Union, 93(16), 163–163. </p>

WaterML2.0: development of an open standard for hydrological time-series data exchange

2013 ◽  
Vol 16 (2) ◽  
pp. 425-446 ◽  
Author(s):  
P. Taylor ◽  
S. Cox ◽  
G. Walker ◽  
D. Valentine ◽  
P. Sheahan
Keyword(s):  
Time Series ◽  
Working Group ◽  
Data Exchange ◽  
Time Series Data ◽  
Series Data ◽  
Observation Data ◽  
Environmental Reporting ◽  
Open Standard ◽  
Terrestrial Water ◽  
Future Work

The increasing global demand on freshwater is resulting in nations improving their terrestrial water monitoring and reporting systems to better understand the availability, and quality, of this valuable resource. A barrier to this is the inability for stakeholders to share information relating to water observations data: traditional hydrological information systems have relied on internal custom data formats to exchange data, leading to issues in data integration and exchange. Organisations are looking to information standards to assist in data exchange, integration and interpretation to lower costs in use, and re-use, of monitoring data. The WaterML2.0 Standards Working Group (SWG), working within the Open Geospatial Consortium (OGC) and in cooperation with the joint OGC-World Meteorological Organisation (WMO) Hydrology Domain Working Group (HDWG), has developed an open standard for the exchange of water observation data. The focus of the standard is time-series data, commonly used for hydrological applications such as flood forecasting, environmental reporting and hydrological infrastructure, where a lack of standards inhibits efficient re-use and automation. This paper describes the development methodology and principles of WaterML2.0, key parts of its information model, implementation scenarios, evaluation and future work. WaterML2.0 was adopted by the OGC as an official standard in September 2012.

A Novel Approach to Time Series Forecasting Using Model-Free Adaptive Control Framework

2020 ◽  
Author(s):  
Meenakshi Narayan ◽  
Ann Majewicz Fey
Keyword(s):  
Time Series ◽  
Adaptive Control ◽  
Real Time ◽  
Time Series Data ◽  
Mean Squared Error ◽  
Force Sensor ◽  
Sensor Data ◽  
Series Data ◽  
Model Free ◽  
Control Framework

Abstract Sensor data predictions could significantly improve the accuracy and effectiveness of modern control systems; however, existing machine learning and advanced statistical techniques to forecast time series data require significant computational resources which is not ideal for real-time applications. In this paper, we propose a novel forecasting technique called Compact Form Dynamic Linearization Model-Free Prediction (CFDL-MFP) which is derived from the existing model-free adaptive control framework. This approach enables near real-time forecasts of seconds-worth of time-series data due to its basis as an optimal control problem. The performance of the CFDL-MFP algorithm was evaluated using four real datasets including: force sensor readings from surgical needle, ECG measurements for heart rate, and atmospheric temperature and Nile water level recordings. On average, the forecast accuracy of CFDL-MFP was 28% better than the benchmark Autoregressive Integrated Moving Average (ARIMA) algorithm. The maximum computation time of CFDL-MFP was 49.1ms which was 170 times faster than ARIMA. Forecasts were best for deterministic data patterns, such as the ECG data, with a minimum average root mean squared error of (0.2±0.2).

A hybrid fuzzy quantum time series and linear programming model: Special application on TAIEX index dataset

2019 ◽  
Vol 34 (25) ◽  
pp. 1950201 ◽  
Author(s):  
Pritpal Singh ◽  
Gaurav Dhiman ◽  
Sen Guo ◽  
Ritika Maini ◽  
Harsimran Kaur ◽  
...  
Keyword(s):  
Time Series ◽  
Linear Programming ◽  
Real Time ◽  
Time Series Data ◽  
Programming Model ◽  
Series Data ◽  
Quantum Model ◽  
Quantum Time ◽  
Degree Of Membership ◽  
Quantum Approach

The supremacy of quantum approach is able to provide the solutions which are not practically feasible on classical machines. This paper introduces a novel quantum model for time series data which depends on the appropriate length of intervals. In this study, the effects of these drawbacks are elaborately illustrated, and some significant measures to remove them are suggested, such as use of degree of membership along with mid-value of the interval. All these improvements signify the effective results in case of quantum time series, which are verified and validated with real-time datasets.

scholarly journals AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

2020 ◽  
Vol 496 (1) ◽  
pp. 629-637
Author(s):  
Ce Yu ◽  
Kun Li ◽  
Shanjiang Tang ◽  
Chao Sun ◽  
Bin Ma ◽  
...  
Keyword(s):  
Time Series ◽  
High Performance ◽  
Large Scale ◽  
Extrasolar Planets ◽  
Time Series Data ◽  
Series Data ◽  
Data Sets ◽  
Observation Data ◽  
Data Volume ◽  
And Performance

ABSTRACT Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analysing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FITS) files or data bases, match each item to determine which object it belongs to, and finally produce time series data sets. To support the high-performance parallel processing of large-scale data sets, AstroCatR uses the extract-transform-load (ETL) pre-processing module to create sky zone files and balance the workload. The matching module uses the overlapped indexing method and an in-memory reference table to improve accuracy and performance. The output of AstroCatR can be stored in CSV files or be transformed other into formats as needed. Simultaneously, the module-based software architecture ensures the flexibility and scalability of AstroCatR. We evaluated AstroCatR with actual observation data from The three Antarctic Survey Telescopes (AST3). The experiments demonstrate that AstroCatR can efficiently and flexibly reconstruct all time series data by setting relevant parameters and configuration files. Furthermore, the tool is approximately 3× faster than methods using relational data base management systems at matching massive catalogues.

Sign in / Sign up

Export Citation Format

Share Document