Big Data Workflows: Locality-Aware Orchestration Using Software Containers

The emergence of the edge computing paradigm has shifted data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructures. Therefore, data processing solutions must consider data locality to reduce the performance penalties from data transfers among remote data centres. Existing big data processing solutions provide limited support for handling data locality and are inefficient in processing small and frequent events specific to the edge environments. This article proposes a novel architecture and a proof-of-concept implementation for software container-centric big data workflow orchestration that puts data locality at the forefront. The proposed solution considers the available data locality information, leverages long-lived containers to execute workflow steps, and handles the interaction with different data sources through containers. We compare the proposed solution with Argo workflows and demonstrate a significant performance improvement in the execution speed for processing the same data units. Finally, we carry out experiments with the proposed solution under different configurations and analyze individual aspects affecting the performance of the overall solution.

A Step Change in the Digital Oilfield Arena: Cloud Computing and Workflow Integration for Production Operations Solutions

Digital oilfield applications have been implemented in numerous operating companies to streamline processes and automate workflows to optimize oil and gas production in real-time. These applications are mostly deployed using traditional on-premises systems; where maintenance, accessibility and scalability serves as a major bottleneck for an efficient outcome. In addition to this challenge, the sector still faces limitations in data integration from disparate data sources, liberation of consolidated data for consumption and cross domain workflow orchestration of that data. The dimensional change brought by digital transformation strategies has paved a path for the Cloud- based solutions, which have recently gained momentum in the oil and gas industry pertaining to their wider accessibility, simpler customization, greater system stability and scalability to support larger amount of data in a performant way. To address the challenges mentioned earlier, we have embarked on a journey with Production Data Foundation which brings together production and equipment data from across an organization. In this paper, we will highlight how Production Data Foundation, hosted on the cloud, provides the underlying infrastructure, services, interfaces required to support and unify production data ingestion, workflow orchestration, and through the alignment of the common domain and digital concepts, improve collaboration between people in distinct roles, such as production engineers, reservoir engineers, drilling engineers, deployment engineers, software developers, data scientists, architects, and subject matter experts (SME) working with production operations products and solutions.

Metagenomics workflow for hybrid assembly, differential coverage binning, metatranscriptomics and pathway analysis (MUFFIN)

Metagenomics has redefined many areas of microbiology. However, metagenome-assembled genomes (MAGs) are often fragmented, primarily when sequencing was performed with short reads. Recent long-read sequencing technologies promise to improve genome reconstruction. However, the integration of two different sequencing modalities makes downstream analyses complex. We, therefore, developed MUFFIN, a complete metagenomic workflow that uses short and long reads to produce high-quality bins and their annotations. The workflow is written by using Nextflow, a workflow orchestration software, to achieve high reproducibility and fast and straightforward use. This workflow also produces the taxonomic classification and KEGG pathways of the bins and can be further used for quantification and annotation by providing RNA-Seq data (optionally). We tested the workflow using twenty biogas reactor samples and assessed the capacity of MUFFIN to process and output relevant files needed to analyze the microbial community and their function. MUFFIN produces functional pathway predictions and, if provided de novo metatranscript annotations across the metagenomic sample and for each bin. MUFFIN is available on github under GNUv3 licence: https://github.com/RVanDamme/MUFFIN.

Metagenomics workflow for hybrid assembly, differential coverage binning, transcriptomics and pathway analysis (MUFFIN)

Metagenomics has redefined many areas of microbiology. However, metagenome-assembled genomes (MAGs) are often fragmented, primarily when sequencing was performed with short reads. Recent long-read sequencing technologies promise to improve genome reconstruction. However, the integration of two different sequencing modalities makes downstream analyses complex. We, therefore, developed MUFFIN, a complete metagenomic workflow that uses short and long reads to produce high-quality bins and their annotations. The workflow is written by using Nextflow, a workflow orchestration software, to achieve high reproducibility and fast and straightforward use. This workflow also produces the taxonomic classification and KEGG pathways of the bins and can be further used by providing RNA-Seq data (optionally) for quantification and annotation. We tested the workflow using twenty biogas reactor samples and assessed the capacity of MUFFIN to process and output relevant files needed to analyze the microbial community and their function. MUFFIN produces functional pathway predictions and if provided de novo transcript annotations across the metagenomic sample and for each bin.Author SummaryRVD did the development and design of MUFFIN and wrote the first draft; BM and EBR did the critical reading and correction of the manuscript; MH did the critical reading of the manuscript and the general adjustments for the metagenomic workflow; AV did the critical reading of the manuscript and adjustments for the taxonomic classifications. CB supervised the project, did the workflow design, helped with the implementation, and revised the manuscript.

BioExcel Building Blocks, a software library for interoperable biomolecular simulation workflows

In the recent years, the improvement of software and hardware performance has made biomolecular simulations a mature tool for the study of biological processes. Simulation length and the size and complexity of the analyzed systems make simulations both complementary and compatible with other bioinformatics disciplines. However, the characteristics of the software packages used for simulation have prevented the adoption of the technologies accepted in other bioinformatics fields like automated deployment systems, workflow orchestration, or the use of software containers. We present here a comprehensive exercise to bring biomolecular simulations to the “bioinformatics way of working”. The exercise has led to the development of the BioExcel Building Blocks (BioBB) library. BioBB’s are built as Python wrappers to provide an interoperable architecture. BioBB’s have been integrated in a chain of usual software management tools to generate data ontologies, documentation, installation packages, software containers and ways of integration with workflow managers, that make them usable in most computational environments.