Portable runtime environments for Python-based FMUs: Adding Docker support to UniFMU

Serverless computing has introduced scalable event-driven processing in Cloud infrastructures. However, it is not trivial for multimedia processing to benefit from the elastic capabilities featured by serverless applications. To this aim, this paper introduces the evolution of a framework to support the execution of customized runtime environments in AWS Lambda in order to accommodate workloads that do not satisfy its strict computational requirements: increased execution times and the ability to use GPU-based resources. This has been achieved through the integration of AWS Batch, a managed service to deploy virtual elastic clusters for the execution of containerized jobs. In addition, a Functions Definition Language (FDL) is introduced for the description of data-driven workflows of functions. These workflows can simultaneously leverage both AWS Lambda for the highly-scalable execution of short jobs and AWS Batch, for the execution of compute-intensive jobs that can profit from GPU-based computing. To assess the developed open-source framework, we executed a case study for efficient serverless video processing. The workflow automatically generates subtitles based on the audio and applies GPU-based object recognition to the video frames, thus simultaneously harnessing different computing services. This allows for the creation of cost-effective highly-parallel scale-to-zero serverless workflows in AWS.

Download Full-text

SpinSPJ: a novel NMR scripting system to implement artificial intelligence and advanced applications

BMC Bioinformatics ◽

10.1186/s12859-021-04492-y ◽

2021 ◽

Vol 22 (1) ◽

Author(s):

Zao Liu ◽

Zhiwei Chen ◽

Kan Song

Keyword(s):

Data Processing ◽

Emerging Technologies ◽

Main Program ◽

Multivariate Statistical ◽

Instrument Control ◽

Nmr Methods ◽

User Friendly ◽

Java Native Interface ◽

Runtime Environments ◽

Data Processing Methods

Abstract Background Software for nuclear magnetic resonance (NMR) spectrometers offer general functionality of instrument control and data processing; these applications are often developed with non-scripting languages. NMR users need to flexibly integrate rapidly developing NMR applications with emerging technologies. Scripting systems offer open environments for NMR users to write custom programs. However, existing scripting systems have limited capabilities for both extending the functionality of NMR software’s non-script main program and using advanced native script libraries to support specialized application domains (e.g., biomacromolecules and metabolomics). Therefore, it is essential to design a novel scripting system to address both of these needs. Result Here, a novel NMR scripting system named SpinSPJ is proposed. It works as a plug-in in the Java based NMR spectrometer software SpinStudioJ. In the scripting system, both Java based NMR methods and original CPython based libraries are supported. A module has been developed as a bridge to integrate the runtime environments of Java and CPython. The module works as an extension in the CPython environment and interacts with Java via the Java Native Interface. Leveraging this bridge, Java based instrument control and data processing methods of SpinStudioJ can be called with the CPython style. Compared with traditional scripting systems, SpinSPJ better supports both extending the non-script main program and implementing advanced NMR applications with a rich variety of script libraries. NMR researchers can easily call functions of instrument control and data processing as well as developing complex functionality (such as multivariate statistical analysis, deep learning, etc.) with CPython native libraries. Conclusion SpinSPJ offers a user-friendly environment to implement custom functionality leveraging its powerful basic NMR and rich CPython libraries. NMR applications with emerging technologies can be easily integrated. The scripting system is free of charge and can be downloaded by visiting http://www.spinstudioj.net/spinspj.

Download Full-text

Standalone containers with ATLAS offline software

EPJ Web of Conferences ◽

10.1051/epjconf/202024507010 ◽

2020 ◽

Vol 245 ◽

pp. 07010

Author(s):

Marcelo Vogel ◽

Mikhail Borodin ◽

Alessandra Forti ◽

Lukas Heinrich

Keyword(s):

File Systems ◽

Network Connectivity ◽

Data Reconstruction ◽

Atlas Experiment ◽

Network Connection ◽

Software Packages ◽

Processing Data ◽

Runtime Environment ◽

Detector Simulation ◽

Runtime Environments

This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can encapsulate software packages inside complete file systems. The deployment of offline releases via containers removes the interdependence between the runtime environment needed for job execution and the configuration of the computing nodes at the sites. Docker or Singularity would provide a uniform runtime environment for the grid, HPCs and for a variety of opportunistic resources. Additionally, releases may be supplemented with a detector’s conditions data, thus removing the need for network connectivity at computing nodes, which is normally quite restricted for HPCs. In preparation to achieve this goal, we have built Docker and Singularity images containing single full releases of ATLAS software for running detector simulation and reconstruction jobs in runtime environments without a network connection. Unlike similar efforts to produce containers by packing all possible dependencies of every possible workflow into heavy images (≈ 200GB), our approach is to include only what is needed for specific workflows and to manage dependencies efficiently via software package managers. This approach leads to more stable packaged releases where the dependencies are clear and the resulting images have more portable sizes ( 16GB). In an effort to cover a wider variety of workflows, we are deploying images that can be used in raw data reconstruction. This is particularly challenging due to the high database resource consumption during the access to the experiment’s conditions payload when processing data. We describe here a prototype pipeline in which images are provisioned only with the conditions payload necessary to satisfy the jobs’ requirements. This database-on-demand approach would keep images slim, portable and capable of supporting various workflows in a standalone fashion in environments with no network connectivity.

Download Full-text

Event Processing in Web Service Runtime Environments

Principles and Applications of Distributed Event-Based Systems - Advances in Systems Analysis, Software Engineering, and High Performance Computing ◽

10.4018/978-1-60566-697-6.ch012 ◽

2010 ◽

pp. 284-306 ◽

Cited By ~ 1

Author(s):

Anton Michlmayr ◽

Philipp Leitner ◽

Florian Rosenberg ◽

Schahram Dustdar

Keyword(s):

Web Service ◽

Service Discovery ◽

Historical Event ◽

Event Processing ◽

Service Oriented Architectures ◽

Loosely Coupled ◽

Service Invocation ◽

Runtime Environment ◽

Service Oriented ◽

Runtime Environments

Service-oriented Architectures (SOA) and Web services have received a lot of attention from both industry and academia. Services as the core entities of every SOA are changing regularly based on various reasons. This poses a clear problem in distributed environments since service providers and consumers are generally loosely coupled. Using the publish/subscribe style of communication service consumers can be notified when such changes occur. In this chapter, we present an approach that leverages event processing mechanisms for Web service runtime environments based on a rich event model and different event visibilities. Our approach covers the full service lifecycle, including runtime information concerning service discovery and service invocation, as well as Quality of Service attributes. Furthermore, besides subscribing to events of interest, users can also search in historical event data. We show how this event notification support was integrated into our service runtime environment VRESCo and give some usage examples in an application context.

Download Full-text