Pycheron: A Python-Based Seismic Waveform Data Quality Control Software Package

Author(s):  
Katherine Anderson Aur ◽  
Jessica Bobeck ◽  
Anthony Alberti ◽  
Phillip Kay

Abstract Supplementing an existing high-quality seismic monitoring network with openly available station data could improve coverage and decrease magnitudes of completeness; however, this can present challenges when varying levels of data quality exist. Without discerning the quality of openly available data, using it poses significant data management, analysis, and interpretation issues. Incorporating additional stations without properly identifying and mitigating data quality problems can degrade overall monitoring capability. If openly available stations are to be used routinely, a robust, automated data quality assessment for a wide range of quality control (QC) issues is essential. To meet this need, we developed Pycheron, a Python-based library for QC of seismic waveform data. Pycheron was initially based on the Incorporated Research Institutions for Seismology’s Modular Utility for STAtistical kNowledge Gathering but has been expanded to include more functionality. Pycheron can be implemented at the beginning of a data processing pipeline or can process stand-alone data sets. Its objectives are to (1) identify specific QC issues; (2) automatically assess data quality and instrumentation health; (3) serve as a basic service that all data processing builds on by alerting downstream processing algorithms to any quality degradation; and (4) improve our ability to process orders of magnitudes more data through performance optimizations. This article provides an overview of Pycheron, its features, basic workflow, and an example application using a synthetic QC data set.

2020 ◽  
Author(s):  
Manuela Köllner ◽  
Mayumi Wilms ◽  
Anne-Christin Schulz ◽  
Martin Moritz ◽  
Katrin Latarius ◽  
...  

<p>Reliable data are the basis for successful research and scientific publishing. Open data policies assure the availability of publicly financed field measurements to the public, thus to all interested scientists. However, the variety of data sources and the availability or lack of detailed metadata cause a huge effort for each scientist to decide if the data are usable for their own research topic or not. Data end-user communities have different requirements in metadata details and data handling during data processing. For data providing institutes or agencies, these needs are essential to know, if they want to reach a wide range of end-user communities.</p><p>The Federal Maritime and Hydrographic Agency (BSH, Bundesamt für Seeschifffahrt und Hydrographie, Hamburg, Germany) is collecting a large variety of field data in physical and chemical oceanography, regionally focused on the North Sea, Baltic Sea, and North Atlantic. Data types vary from vertical profiles, time-series, underway measurements as well as real-time or delayed-mode from moored or ship-based instruments. Along other oceanographic data, the BSH provides all physical data via the German Oceanographic Data Center (DOD). It is crucial to aim for a maximum in reliability of the published data to enhance the usage especially in the scientific community.</p><p>Here, we present our newly established data processing and quality control procedures using agile project management and workflow techniques, and outline their implementation into metadata and accompanied documentation. To enhance the transparency of data quality control, we will apply a detailed quality flag along with the common data quality flag. This detailed quality flag, established by Mayumi Wilms within the research project RAVE Offshore service (research at alpha ventus) enables data end-users to review the result of several individual quality control checks done during processing and thus to identify easily if the data are usable for their research.</p>


2021 ◽  
Author(s):  
Carlo Cauzzi ◽  
Jarek Bieńkowski ◽  
Susana Custódio ◽  
Sebastiano D'Amico ◽  
Christos Evangelidis ◽  
...  

<p>ORFEUS (Observatories and Research Facilities for European Seismology; http://orfeus-eu.org/) is a non-profit foundation that promotes observational seismology in the Euro-Mediterranean area through the collection, archival and distribution of seismic waveform data, metadata, and closely related services and products. The data and services are collected or developed at national level by more than 60 contributing Institutions in Pan-Europe and further enhanced, integrated, standardized, homogenized and promoted through ORFEUS. Among the goals of ORFEUS are: (a) the development and coordination of waveform data products; (b) the coordination of a European data distribution system, and the support for seismic networks in archiving and exchanging digital seismic waveform data; (c) the encouragement of the adoption of best practices for seismic network operation, data quality control and FAIR data management; (d) the promotion of open access to seismic waveform data, products and services for the broader Earth science community. These goals are achieved through the development and maintenance of services targeted to a broad community of seismological data users, ranging from earth scientists to earthquake engineering practitioners. Two Service Management Committees (SMCs) are consolidated within ORFEUS devoted to managing, operating and developing (with the support of one or more Infrastructure Development Groups): (i) the European Integrated Data Archive (EIDA; https://www.orfeus-eu.org/data/eida/); and (ii) the European Strong-Motion databases (SM; https://www.orfeus-eu.org/data/strong/). New emerging groups within ORFEUS are focused on mobile pools and computational seismology. ORFEUS services currently provide access to the waveforms acquired by ~ 14,500 stations, including dense temporary experiments, with strong emphasis on open, high-quality data. Contributing to ORFEUS data archives means benefitting from long-term archival, state-of-the-art quality control, improved access, increased usage, and community participation. Access to data and products is ensured through state-of-the-art information and communication technologies, with strong emphasis on federated web services that considerably improve seamless user access to data gathered and/or distributed by the various ORFEUS institutions. Web services also facilitate the automation of downstream products. Particular attention is paid to adopting clear policies and licenses, and acknowledging the crucial role played by data providers / owners, who are part of the ORFEUS community. There are significant efforts by ORFEUS participating Institutions to enhance the existing services to tackle the challenges posed by the Big Data Era, with emphasis on data quality, improved user experience, and implementation of strategies for scalability, high-volume data access and archival. ORFEUS data and services are assessed and improved through the technical and scientific feedback of a User Advisory Group (UAG), which comprises European Earth scientists with expertise on a broad range of disciplines. All ORFEUS services are developed in coordination with EPOS and are largely integrated in the EPOS Data Access Portal, as ORFEUS is one of the founding Parties and fundamental contributors of the EPOS Thematic Core Service for Seismology (https://www.epos-eu.org/tcs/seismology). In this contribution, we selectively present the activities of ORFEUS, with the main aims of facilitating seismological data discovery and encouraging open data sharing and integration, as well as promoting best practice in observational seismology.</p>


Author(s):  
F. D. Vescovi ◽  
T. Lankester ◽  
E. Coleman ◽  
G. Ottavianelli

The Copernicus Space Component Data Access system (CSCDA) incorporates data contributions from a wide range of satellite missions. Through EO data handling and distribution, CSCDA serves a set of Copernicus Services related to Land, Marine and Atmosphere Monitoring, Emergency Management and Security and Climate Change. <br><br> The quality of the delivered EO products is the responsibility of each contributing mission, and the Copernicus data Quality Control (CQC) service supports and complements such data quality control activities. The mission of the CQC is to provide a service of quality assessment on the provided imagery, to support the investigation related to product quality anomalies, and to guarantee harmonisation and traceability of the quality information. <br><br> In terms of product quality control, the CQC carries out analysis of representative sample products for each contributing mission as well as coordinating data quality investigation related to issues found or raised by Copernicus users. Results from the product analysis are systematically collected and the derived quality reports stored in a searchable database. <br><br> The CQC service can be seen as a privileged focal point with unique comparison capacities over the data providers. The comparison among products from different missions suggests the need for a strong, common effort of harmonisation. Technical terms, definitions, metadata, file formats, processing levels, algorithms, cal/val procedures etc. are far from being homogeneous, and this may generate inconsistencies and confusion among users of EO data. <br><br> The CSCDA CQC team plays a significant role in promoting harmonisation initiatives across the numerous contributing missions, so that a common effort can achieve optimal complementarity and compatibility among the EO data from multiple data providers. This effort is done in coordination with important initiatives already working towards these goals (e.g. INSPIRE directive, CEOS initiatives, OGC standards, QA4EO etc.). <br><br> This paper describes the main actions being undertaken by CQC to encourage harmonisation among space-based EO systems currently in service.


2020 ◽  
Author(s):  
Carlo Cauzzi ◽  
Jarek Bieńkowski ◽  
Susana Custódio ◽  
Christos Evangelidis ◽  
Philippe Guéguen ◽  
...  

&lt;p&gt;ORFEUS (Observatories and Research Facilities for European Seismology) is a non-profit foundation that promotes seismology in the Euro-Mediterranean area through the collection, archival and distribution of seismic waveform data, metadata and closely related products. The data and services are collected or developed at national level by more than 60 contributing Institutions in Pan-Europe and further developed, integrated, standardized, homogenized and promoted through ORFEUS. Among the goals of ORFEUS are: (a) the development and coordination of waveform data products; (b) the coordination of a European data distribution system, and the support for seismic networks in archiving and exchanging digital seismic waveform data; (c) the encouragement of the adoption of best practices for seismic network operation, data quality control and data management; (d) the promotion of open access to seismic waveform data, products and services for the broader Earth science community.&amp;#160; These goals are achieved through the development and maintenance of services targeted to a broad community of seismological data users, ranging from earth scientists to earthquake engineering practitioners. Two Service Management Committees (SMCs) are consolidated within ORFEUS devoted to managing, operating and developing (with the support of one or more Infrastructure Development Groups): (i) the European Integrated waveform Data Archive (EIDA; https://www.orfeus-eu.org/data/eida/); and (ii) the European Strong-Motion databases (SM; https://www.orfeus-eu.org/data/strong/). A new SMC is being formed to represent the community of European mobile pools. Products and services for computational seismologists are also considered for integration in the ORFEUS domain. ORFEUS services currently provide access to the waveforms acquired by ~ 10,000 stations in Pan-Europe, including dense temporary experiments, with strong emphasis on open, high-quality data. Contributing to ORFEUS data archives means long-term archival, state-of-the-art quality control, improved access and increased&amp;#160; usage. Access to data and products is ensured through state-of-the-art information and communications technologies, with strong emphasis on federated web services that considerably improve seamless user access to data gathered and/or distributed by ORFEUS institutions. The web services also facilitate the automation of downstream products. Particular attention is paid to adopting clear policies and licences, and acknowledging the crucial role played by data providers / owners, who are part of the ORFEUS community. There are significant efforts by ORFEUS participating Institutions to enhance the existing services to tackle the challenges posed by the Big Data Era, with emphasis on data quality, improved user experience, and implementation of strategies for scalability, high-volume data access and archival. ORFEUS data and services are assessed and improved through the technical and scientific feedback of a User Advisory Group (UAG), comprised of European Earth scientists with expertise encompassing a broad range of disciplines. All ORFEUS services are developed in coordination with EPOS and are largely integrated in the EPOS Data Access Portal. ORFEUS is one of the founding Parties and fundamental pillars of EPOS Seismology. This contribution presents the current products and services of ORFEUS and introduces the planned key future activities. We aim at stimulating Community feedback about the current and planned ORFEUS strategies.&lt;/p&gt;


2004 ◽  
Vol 50 (11) ◽  
pp. 51-58 ◽  
Author(s):  
S. Ciavatta ◽  
R. Pastres ◽  
Z. Lin ◽  
M.B. Beck ◽  
C. Badetti ◽  
...  

In the context of monitoring water quality in natural ecosystems in real time, on-line data quality control is a very important issue for effective system surveillance and for optimizing maintenance of the monitoring network. This paper presents some applications of recursive state-parameter estimation algorithms to real-time detection of signal drift in high-frequency observations. Two continuous-discrete recursive estimation schemes, namely the Extended Kalman Filter and the Recursive Prediction Error algorithm, were applied to assuring the quality of the dissolved oxygen (DO) time series, as obtained from the Lagoon of Venice (Italy) during August 2002, through the real-time monitoring network of the Magistrato alle Acque (the Venice Water Authority). Results demonstrate the effectiveness of the methodology in early detection of a probable drift in the DO signal. Comparison of these results with those obtained from the application of a related recursive scheme (a Dynamic Linear Regression procedure) suggests the strong benefits of approaching the problem of on-line data quality control with several (not merely a single) independent such estimation methods.


2020 ◽  
Author(s):  
Matthias Maeyens ◽  
Brianna Pagán ◽  
Piet Seuntjens ◽  
Bino Maiheu ◽  
Nele Desmet ◽  
...  

&lt;p&gt;In recent years, extend periods of drought have been affecting the water quality and availability in &amp;#160;the Flanders region in Belgium. Especially the coastal region experienced an increased salinization of ground and surface water. The Flemish government therefore decided to invest in a dense IoT water quality monitoring network aiming to deploy 2500 water quality sensors &amp;#160;primarily in surface water but also in ground water and sewers. The goal of this &quot;Internet of Water&quot; project is to establish an operational state of the art monitoring and prediction system in support of future water policy in Flanders.&amp;#160;&lt;/p&gt;&lt;p&gt;Since Flanders is a relatively small region (13,522&amp;#160;km&amp;#178;), placing this many sensors will result in one of the most dense surface water quality sensor networks in the world. Each sensor will continuously measure several indicators of water quality and transmit the data wirelessly. This allows us to continuously monitor the water quality and build a big enough data set to be able to use a more data driven approach to predicting changes&amp;#160; in water quality. However, as with any sensor system, the quality of the data can vary in time due to problems with the sensors, incorrect calibration or unforeseen issues. Real-time data quality control is crucial to prevent unsound decisions due to faulty data.&lt;/p&gt;&lt;p&gt;This contribution will give a general overview of the network and it&amp;#8217;s specifications, but mainly focus on the implementation of the data stream as well as methods that are implemented to guarantee good data quality. More specifically the architecture and setup of a real-time data quality control system is described. Which will add quality control flags to measurements. &amp;#160;This system is&amp;#160; integrated with the NGSI API introduced by FIWARE, which forces us to make specific design decisions to acommodate to the NGSI API.&lt;/p&gt;


Author(s):  
Antonella D. Pontoriero ◽  
Giovanna Nordio ◽  
Rubaida Easmin ◽  
Alessio Giacomel ◽  
Barbara Santangelo ◽  
...  

Sign in / Sign up

Export Citation Format

Share Document