Connecting data streams with On-Demand Services in the Alpine Environmental Data Analysis Centre

2020 ◽  
Author(s):  
Johannes Munke ◽  
Alexander Götz ◽  
Helmut Heller ◽  
Stephan Hachinger ◽  
Dominik Laux ◽  
...  
Keyword(s):  
Data Analysis ◽  
Stream Processing ◽  
Application Programming Interface ◽  
Scientific Data ◽  
Environmental Data ◽  
Message Queue ◽  
On Demand ◽  
Research Activities ◽  
Rest Api ◽  
Center Platform

<div> <p>The AlpEnDAC (Alpine Environmental Data Analysis Center) platform (www.alpendac.eu) aims to collect scientific data measured on different high-altitude research stations in the alpine region and beyond. It provides research data management, analysis and simulation services and supports the research activities of the VAO (Virtual Alpine Observatory) community.</p> <p>With funding from the Bavarian State Ministry of the Environment and Consumer Protection, a new development cycle of the platform was launched in 2019. Novel components for Computing on Demand (CoD), Service on Demand (SoD) and Operating on Demand (OoD) will be integrated into the system. These will help to implement a near-real-time (NRT) decision support for the scientist during measurement processes and a better control of the measurement process.</p> <p>In this work, the authors present a stream processing architecture to couple the new CoD, SoD and OoD components. Data from measurements (or also simulations) are normally ingested via a representational state transfer application programming interface (REST API) into the AlpEnDAC system. Before such data are stored in the data base, they will be run through a central stream processing engine, based on a message queue (e.g. Apache Kafka) and a series of specialized workers to process the data. A rule engine and analytics tools are connected to this engine and allow the automatic triggering of, e.g., measurements, HPC simulations, or evaluation and notification services in NRT. The services will be usable and configurable, as much as possible, via the AlpEnDAC web portal where also certain measurement device settings can be adjusted. With these developments, we want to make environmental scientists profit from NRT data collection and processing, as it is already an everyday tool, e.g., in the Internet-of-Things sector and in commercial applications.</p> </div>

Leveraging Web 2.0 technologies to add value to the IUPAC Solubility Data Series: development of a REST style website and application programming interface (API)

2015 ◽  
Vol 87 (11-12) ◽  
pp. 1127-1137
Author(s):  
Stuart J. Chalk
Keyword(s):  
Application Programming Interface ◽  
Scientific Data ◽  
Lessons Learned ◽  
Data Series ◽  
Representational State Transfer ◽  
Database Construction ◽  
Series Development ◽  
Application Programming ◽  
Rest Api ◽  
Program Interface

AbstractThis paper details an approach to re-purposing scientific data as presented on a web page for the sole purpose of making the data more available for searching and integration into other websites. Data ‘scraping’ is used to extract metadata from a set of pages on the National Institute of Standards and Technology (NIST) website, clean, organize and store the metadata in a MySQL database. The metadata is then used to create a new website at the authors institution using the CakePHP framework to create a representational state transfer (REST) style application program interface (API). The processes used for website analysis, schema development, database construction, metadata scraping, REST API development, and remote data integration are discussed. Lessons learned and tips and tricks on how to get the most out of the process are also included.

Stream processing for near real-time scientific data analysis

2016 ◽  
Author(s):  
Jong Youl Choi ◽  
Tahsin Kurc ◽  
Jeremy Logan ◽  
Matthew Wolf ◽  
Eric Suchyta ◽  
...  
Keyword(s):  
Data Analysis ◽  
Real Time ◽  
Stream Processing ◽  
Scientific Data ◽  
Scientific Data Analysis

scholarly journals The eReefs data brokering layer for hydrological and environmental data

2015 ◽  
Vol 18 (2) ◽  
pp. 152-167 ◽  
Author(s):  
Jonathan Yu ◽  
Benjamin Leighton ◽  
Nicholas Car ◽  
Shane Seaton ◽  
Jonathan Hodge
Keyword(s):  
Application Programming Interface ◽  
Environmental Data ◽  
Measurement Properties ◽  
Data Service ◽  
Information Platform ◽  
Data Services ◽  
Research Activities ◽  
Virtual Laboratories ◽  
Application Programming ◽  
Programming Interface

The environmental sciences are witnessing a data revolution as large amounts of data are being made available at an increasing rate. Many datasets are being published through operational monitoring programs, research activities and global earth observation virtual laboratories. An important aspect is the ability to query relevant metadata which can potentially provide useful information to discover, access and interpret environmental datasets, information about the data providers themselves, data services, data encodings, observation and measurement properties and data service endpoints. However, support for producing and accessing metadata descriptions in a flexible, extensible, easily integrated and easily discovered manner is lacking as current methods require interpreting multiple standards and formalisms. In this paper, we propose components to streamline discovery and access of hydrological and environmental data: a Data Provider Node ontology (DPN-O) which allows precise descriptions to be captured about datasets, data services and their interfaces; and a Data Brokering Layer which provides an Application Programming Interface (API) for registering metadata for discovery and query of registered DPN datasets. We discuss this work in the context of the eReefs project which is developing an integrated information platform for discovery and visualization of observational and modelled data of the Great Barrier Reef.

The Alpine Environmental Data Analysis Centre (www.alpendac.eu) – A Component of the Virtual Alpine Observatory (VAO) (www.vao.bayern.de)

2020 ◽  
Author(s):  
Michael Bittner ◽  
Dominik Laux ◽  
Oleg Goussev ◽  
Sabine Wüst ◽  
Jana Handschuih ◽  
...  
Keyword(s):  
Data Analysis ◽  
Data Management ◽  
Heterogeneous Data ◽  
Research Data ◽  
Environmental Data ◽  
Research Network ◽  
Data Sets ◽  
Measurement Instruments ◽  
On Demand ◽  
Simulation Techniques

<p>The “Alpine Environmental Data Analysis Centre” (AlpEnDAC) is a research data management and analysis platform for research facilities around the Alps and similar mountain ranges. It provides the computational infrastructure for the Virtual Alpine Observatory (VAO), which is a research network of European high-altitude research stations (http://www.vao.bayern.de).</p><p> </p><p>Within the scope of previous work, the platform was developed with the focus on research data and metadata management as well as analysis and simulation tools. It offers the possibility to store and retrieve data securely (data-on-demand), to share it with other scientists and to interpret it with the help of computing-on-demand solutions via a user friendly web-based graphical user interface. The AlpEnDAC allows the analysis and consolidation of heterogeneous data sets from ground-based to satellite instruments.</p><p> </p><p>In a further development phase, launched on 1 August 2019, the existing services of the AlpEnDAC will be supplemented by new components in the fields of user support and quality assurance. Furthermore, the modelling and analysis software portfolio will be extended, focusing on the development of innovative services in the fields of service-on-demand and operating-on-demand as well as the integration of new data sources and measurement instruments.</p><p> </p><p>The AlpEnDAC helps environmental scientists to benefit from modern data management, data analysis, and simulation techniques. The VAO network, now including ten countries (Austria, France, Germany, Georgia, Italy, Norway, Slovenia, Switzerland, Bulgaria, and the Czech Republic) is an ideal and exciting context for developing the AlpEnDAC with researchers.</p><p> </p><p>This project receives funding from the Bavarian State Ministry of the Environment and Consumer Protection.</p>

scholarly journals A Python-oriented environment for climate experiments at scale in the frame of the European Open Science Cloud

2020 ◽  
Author(s):  
Donatello Elia ◽  
Fabrizio Antonio ◽  
Cosimo Palazzo ◽  
Paola Nassisi ◽  
Sofiane Bendoukha ◽  
...  
Keyword(s):  
Data Analysis ◽  
Data Analytics ◽  
Large Scale ◽  
Data Access ◽  
Open Science ◽  
Scientific Data ◽  
Precipitation Trend ◽  
Data Intensive ◽  
Research Activities ◽  
The Eu

<p>Scientific data analysis experiments and applications require software capable of handling domain-specific and data-intensive workflows. The increasing volume of scientific data is further exacerbating these data management and analytics challenges, pushing the community towards the definition of novel programming environments for dealing efficiently with complex experiments, while abstracting from the underlying computing infrastructure. </p><p>ECASLab provides a user-friendly data analytics environment to support scientists in their daily research activities, in particular in the climate change domain, by integrating analysis tools with scientific datasets (e.g., from the ESGF data archive) and computing resources (i.e., Cloud and HPC-based). It combines the features of the ENES Climate Analytics Service (ECAS) and the JupyterHub service, with a wide set of scientific libraries from the Python landscape for data manipulation, analysis and visualization. ECASLab is being set up in the frame of the European Open Science Cloud (EOSC) platform - in the EU H2020 EOSC-Hub project - by CMCC (https://ecaslab.cmcc.it/) and DKRZ (https://ecaslab.dkrz.de/), which host two major instances of the environment. </p><p>ECAS, which lies at the heart of ECASLab, enables scientists to perform data analysis experiments on large volumes of multi-dimensional data by providing a workflow-oriented, PID-supported, server-side and distributed computing approach. ECAS consists of multiple components, centered around the Ophidia High Performance Data Analytics framework, which has been integrated with data access and sharing services (e.g., EUDAT B2DROP/B2SHARE, Onedata), along with the EGI federated cloud infrastructure. The integration with JupyterHub provides a convenient interface for scientists to access the ECAS features for the development and execution of experiments, as well as for sharing results (and the experiment/workflow definition itself). ECAS parallel data analytics capabilities can be easily exploited in Jupyter Notebooks (by means of PyOphidia, the Ophidia Python bindings) together with well-known Python modules for processing and for plotting the results on charts and maps (e.g., Dask, Xarray, NumPy, Matplotlib, etc.). ECAS is also one of the compute services made available to climate scientists by the EU H2020 IS-ENES3 project. </p><p>Hence, this integrated environment represents a complete software stack for the design and run of interactive experiments as well as complex and data-intensive workflows. One class of such large-scale workflows, efficiently implemented through the environment resources, refers to multi-model data analysis in the context of both CMIP5 and CMIP6 (i.e., precipitation trend analysis orchestrated in parallel over multiple CMIP-based datasets).</p>

As-RaD System as a Design Model of the Network Automation Configuration System Based on the REST-API and Django Framework

2020 ◽  
pp. 291-298
Author(s):  
Adian Fatchur Rochim ◽  
Abda Rafi ◽  
Adnan Fauzi ◽  
Kurniawan Teguh Martono
Keyword(s):  
Information Technology ◽  
Application Programming Interface ◽  
Design Model ◽  
Automation System ◽  
Network Configuration ◽  
Web Based ◽  
Representational State Transfer ◽  
Application Programming ◽  
Rest Api ◽  
Python Programming

The use of information technology these days are very high. From business through education activities tend to use this technology most of the time. Information technology uses computer networks for integration and management data. To avoid business problems, the number of network devices installed requires a manageable network configuration for easier maintenance. Traditionally, each of network devices has to be manually configured by network administrators. This process takes time and inefficient. Network automation methods exist to overcome the repetitive process. Design model uses a web-based application for maintenance and automates networking tasks. In this research, the network automation system implemented and built a controller application that used REST API (Representational State Transfer Application Programming Interface) architecture and built by Django framework with Python programming language. The design modeled namely As-RaD System. The network devices used in this research are Cisco CSR1000V because it supports REST API communication to manage its network configuration and could be placed on the server either. The As-RaD System provides 75% faster performance than Paramiko and 92% than NAPALM.

Medicinal importance, pharmacological activities and analytical aspects of canadine in the medicine: An important phytochemical of Hydrastis canadensis

2021 ◽  
Vol 01 ◽  
Author(s):  
Dinesh Kumar Patel
Keyword(s):  
Natural Products ◽  
Data Analysis ◽  
Human Health ◽  
Scientific Information ◽  
Modern Medicine ◽  
Scientific Data ◽  
Pharmacological Activities ◽  
Hydrastis Canadensis ◽  
Biological Importance ◽  
Human Disorders

Background: Natural products and their derived phytochemicals have been used in the medicine and gaining importance in the modern medicine due to their therapeutic potential and health beneficial effect on human disorders. Plenty of herbal drug based products are available in the market and playing an important role in the human health care system due to their health beneficial properties in human being. In the modern age we can find many herbal based products in the market mainly prepared from the natural products and used for the prevention and treatment of various human disorders. Benzylisoquinoline alkaloids are the important class of alkaloidal compounds and the better example are morphine, codeine, sanguinarine, berberine and canadine which are mainly known for their medicinal value in the medicine. Methods: Hydrastis canadensis is the important medicinal plant found to contain a significant amount of canadine, hydrastine and berberine. In the present investigation, numerous scientific databases such as Google, Pubmed, Science direct etc. have been searched to collect the important scientific information of canadine and analyzed to know the health beneficial aspect of canadine in the medicine. All the collected scientific information data’s were analyzed and have been categorized into mainly pharmacological and analytical aspects. Results: From the analysis of the collected scientific information, it was found that Hydrastis canadensis contain significant amount of canadine with many more phytochemical including canadaline, hydrastidine, isohydrastidine etc. Pharmacological activity data analysis revealed the biological importance of Hydrastis canadensis in the medicine for their traditional uses against gastritis, colitis, duodenal ulcers, loss of appetite, liver disease, bile secretion disorder, snake bites and vaginitis in the medicine. However, scientific data analysis of canadine revealed their effectiveness for their acetylcholinesterase inhibitory activity, anti-cancer, anti-microbial, anti-allergic activity and anti-oxidant activity. Different modern analytical tools have been used in the modern medicine for the isolation and quantification of canadine in the Hydrastis canadensis. Conclusion: Present investigation revealed the medicinal importance and pharmacological activities of a canadine in the medicine for the treatment of numerous human health complications. These scientific data will be helpful to the scientist to know the biological importance of canadine in the medicine against various forms of human complications.

Sign in / Sign up

Export Citation Format

Share Document