CBC MAC for Real-Time Data Sources

Purpose The purpose of this paper is to propose a data prediction framework for scenarios which require forecasting demand for large-scale data sources, e.g., sensor networks, securities exchange, electric power secondary system, etc. Concretely, the proposed framework should handle several difficult requirements including the management of gigantic data sources, the need for a fast self-adaptive algorithm, the relatively accurate prediction of multiple time series, and the real-time demand. Design/methodology/approach First, the autoregressive integrated moving average-based prediction algorithm is introduced. Second, the processing framework is designed, which includes a time-series data storage model based on the HBase, and a real-time distributed prediction platform based on Storm. Then, the work principle of this platform is described. Finally, a proof-of-concept testbed is illustrated to verify the proposed framework. Findings Several tests based on Power Grid monitoring data are provided for the proposed framework. The experimental results indicate that prediction data are basically consistent with actual data, processing efficiency is relatively high, and resources consumption is reasonable. Originality/value This paper provides a distributed real-time data prediction framework for large-scale time-series data, which can exactly achieve the requirement of the effective management, prediction efficiency, accuracy, and high concurrency for massive data sources.

Download Full-text

WMO Hydrological Observing System (WHOS) broker: implementation progress and outcomes

10.5194/egusphere-egu2020-14755 ◽

2020 ◽

Author(s):

Enrico Boldrini ◽

Paolo Mazzetti ◽

Stefano Nativi ◽

Mattia Santoro ◽

Fabrizio Papeschi ◽

...

Keyword(s):

Real Time ◽

Data Model ◽

Meteorological Data ◽

Heterogeneous Systems ◽

Data Sources ◽

Technical Documentation ◽

Time Data ◽

Hydrological Data ◽

Real Time Data ◽

Rest Api

The WMO Hydrological Observing System (WHOS) is a service-oriented System of Systems (SoS) linking hydrological data providers and users by enabling harmonized and real time discovery and access functionalities at global, regional, national and local scale. WHOS is being realized through a coordinated and collaborative effort amongst:<ul><li>National Hydrological Services (NHS) willing to publish their data to the benefit of a larger audience,</li> <li>Hydrologists, decision makers, app and portal authors willing to gain access to world-wide hydrological data,</li> <li>ESSI-Lab of CNR-IIA responsible for the WHOS broker component: a software framework in charge of enabling interoperability amongst the distributed heterogeneous systems belonging to data providers (e.g. data publishing services) and data consumers (e.g. web portals, libraries and apps),</li> <li>WMO Commission of Hydrology (CHy) providing guidance to WMO Member countries in operational hydrology, including capacity building, NHSs engagement and coordination of WHOS implementation.</li> </ul>In the last years two additional WMO regional programmes have been targeted to benefit from WHOS, operating as successful applications for others to follow:<ul><li>Plata river basin,</li> <li>Arctic-HYCOS.</li> </ul>Each programme operates with a &#8220;view&#8221; of the whole WHOS, a virtual subset composed only by the data sources that are relevant to its context.WHOS-Plata is currently brokering data sources from the following countries:<ul><li>Argentina (hydrological & meteorological data),</li> <li>Bolivia (meteorological data; hydrological data expected in the near future),</li> <li>Brazil (hydrological & meteorological data),</li> <li>Paraguay (meteorological data; hydrological data in process),</li> <li>Uruguay (hydrological & meteorological data).</li> </ul>WHOS-Arctic is currently brokering data sources from the following countries:<ul><li>Canada (historical and real time data),</li> <li>Denmark (historical data),</li> <li>Finland (historical and real time data),</li> <li>Iceland (historical and real time data),</li> <li>Norway (historical and real time data),</li> <li>Russian (historical and real time data),</li> <li>United States (historical and real time data).</li> </ul>Each data source publishes its data online according to specific hydrological service protocols and/or APIs (e.g. CUAHSI HydroServer, USGS Water Services, FTP, SOAP, REST API, OData, WAF, OGC SOS, &#8230;). Each service protocol and API in turn implies support for a specific metadata and data model (e.g. WaterML, CSV, XML , JSON, USGS RDB, ZRXP, Observations & Measurements, &#8230;).WHOS broker implements mediation and harmonization of all these heterogeneous standards, in order to seamlessly support discovery and access of all the available data to a growing set of data consumer systems (applications and libraries) without any implementation effort for them:<ul><li>52North Helgoland (through SOS v.2.0.0),</li> <li>CUAHSI HydroDesktop (through CUAHSI WaterOneFlow),</li> <li>National Water Institute of Argentina (INA) node.js WaterML client (through CUAHSI WaterOneFlow),</li> <li>DAB JS API (through DAB REST API),</li> <li>USGS GWIS JS API plotting library (through RDB service),</li> <li>R scripts (through R WaterML library),</li> <li>C# applications (through CUAHSI WaterOneFlow),</li> <li>UCAR jOAI (through OAI-PMH/WIGOS metadata).</li> </ul>In particular, the support of WIGOS metadata standard provides a set of observational metadata elements for the effective interpretation of observational data internationally.In addition to metadata and data model heterogeneity, WHOS needs to tackle also semantics heterogeneity. WHOS broker makes use of a hydrology ontology (made available as a SPARQL endpoint) to augment WHOS discovery capabilities (e.g. to obtain translation of a hydrology search parameter in multiple languages).Technical documentation to exercise WHOS broker is already online available, while the official public launch with a dedicated WMO WHOS web portal is expected shortly.

Download Full-text