CloudGT: A High Performance Genome Analysis Toolkit Leveraging Pipeline Optimization on Spark

2018 ◽  
Author(s):  
Anghong Xiao ◽  
Shoubin Dong ◽  
Cheng Liu ◽  
Lingqi Zhang ◽  
Zongze Wu
Keyword(s):  
Genome Analysis ◽  
High Performance ◽  
Genome Analysis Toolkit

scholarly journals Workstation benchmark of Spark Capable Genome Analysis ToolKit 4 Variant Calling

2020 ◽  
Author(s):  
Marcus H. Hansen ◽  
Anita T. Simonsen ◽  
Hans B. Ommen ◽  
Charlotte G. Nyvold
Keyword(s):  
Dna Sequencing ◽  
Genome Analysis ◽  
High Speed ◽  
High Performance ◽  
Variant Calling ◽  
Amplicon Sequencing ◽  
Targeted Sequencing ◽  
Sequencing Analysis ◽  
Genome Analysis Toolkit ◽  
Order Of Magnitude

AbstractBackgroundRapid and practical DNA-sequencing processing has become essential for modern biomedical laboratories, especially in the field of cancer, pathology and genetics. While sequencing turn-over time has been, and still is, a bottleneck in research and diagnostics, the field of bioinformatics is moving at a rapid pace – both in terms of hardware and software development. Here, we benchmarked the local performance of three of the most important Spark-enabled Genome analysis toolkit 4 (GATK4) tools in a targeted sequencing workflow: Duplicate marking, base quality score recalibration (BQSR) and variant calling on targeted DNA sequencing using a modest hyperthreading 12-core single CPU and a high-speed PCI express solid-state drive.ResultsCompared to the previous GATK version the performance of Spark-enabled BQSR and HaplotypeCaller is shifted towards a more efficient usage of the available cores on CPU and outperforms the earlier GATK3.8 version with an order of magnitude reduction in processing time to analysis ready variants, whereas MarkDuplicateSpark was found to be thrice as fast. Furthermore, HaploTypeCallerSpark and BQSRPipelineSpark were significantly faster than the equivalent GATK4 standard tools with a combined ∼86% reduction in execution time, reaching a median rate of ten million processed bases per second, and duplicate marking was reduced ∼42%. The called variants were found to be in close agreement between the Spark and non-Spark versions, with an overall concordance of 98%. In this setup, the tools were also highly efficient when compared execution on a small 72 virtual CPU/18-node Google Cloud cluster.ConclusionIn conclusion, GATK4 offers practical parallelization possibilities for DNA sequence processing, and the Spark-enabled tools optimize performance and utilization of local CPUs. Spark utilizing GATK variant calling is several times faster than previous GATK3.8 multithreading with the same multi-core, single CPU, configuration. The improved opportunities for parallel computations not only hold implications for high-performance cluster, but also for modest laboratory or research workstations for targeted sequencing analysis, such as exome, panel or amplicon sequencing.

scholarly journals PipeMEM: A Framework to Speed Up BWA-MEM in Spark with Low Overhead

Genes ◽  
2019 ◽  
Vol 10 (11) ◽  
pp. 886 ◽  
Author(s):  
Lingqi Zhang ◽  
Cheng Liu ◽  
Shoubin Dong
Keyword(s):  
Genome Analysis ◽  
High Speed ◽  
High Performance ◽  
Genome Alignment ◽  
Single Node ◽  
Genome Data ◽  
Dna Sequence Alignment ◽  
Alignment Tool ◽  
Genome Analysis Toolkit ◽  
Node Solution

(1) Background: DNA sequence alignment process is an essential step in genome analysis. BWA-MEM has been a prevalent single-node tool in genome alignment because of its high speed and accuracy. The exponentially generated genome data requiring a multi-node solution to handle large volumes of data currently remains a challenge. Spark is a ubiquitous big data platform that has been exploited to assist genome alignment in handling this challenge. Nonetheless, existing works that utilize Spark to optimize BWA-MEM suffer from higher overhead. (2) Methods: In this paper, we presented PipeMEM, a framework to accelerate BWA-MEM with lower overhead with the help of the pipe operation in Spark. We additionally proposed to use a pipeline structure and in-memory-computation to accelerate PipeMEM. (3) Results: Our experiments showed that, on paired-end alignment tasks, our framework had low overhead. In a multi-node environment, our framework, on average, was 2.27× faster compared with BWASpark (an alignment tool in Genome Analysis Toolkit (GATK)), and 2.33× faster compared with SparkBWA. (4) Conclusions: PipeMEM could accelerate BWA-MEM in the Spark environment with high performance and low overhead.

INSIDE INDUSTRY

2016 ◽  
Vol 20 (06) ◽  
pp. 45-53
Keyword(s):  
Genome Analysis ◽  
Liquid Biopsy ◽  
High Performance ◽  
Compressed Air ◽  
Opioid Overdose ◽  
Medical Group ◽  
Private Healthcare ◽  
Broad Institute ◽  
Genome Analysis Toolkit ◽  
Unit Dose

APTAR PHARMA Provides Unit-Dose Nasal Spray Technology for Treatment of Opioid Overdose Cloudera, Broad Institute Collaborate on the Next Generation of the Genome Analysis Toolkit Singapore-based Luye Medical Group Completes Acquisition of Healthe Care, Australia's Third Largest Private Healthcare Group FEI Launches Apreo – Industry-Leading Versatile, High-Performance SEM BOGE Publishes New Guide on Specifying Compressed Air for Healthcare Takara Bio USA, Inc. and Integrated DNA Technologies Announce Collaboration to Support Targeted RNA Sequencing Pelican BioThermal Announces Launch of New Asia Headquarters in Singapore A Faster Way to Separate Proteins with Electrophoresis Biosensors Announces Strategic Agreement with Cardinal Health BGI and Clearbridge BioMedics Partner to Develop China CTC Liquid Biopsy Market towards Precision Medicine

scholarly journals From FastQ Data to High‐Confidence Variant Calls: The Genome Analysis Toolkit Best Practices Pipeline

2013 ◽  
Vol 43 (1) ◽  
Author(s):  
Geraldine A. Auwera ◽  
Mauricio O. Carneiro ◽  
Christopher Hartl ◽  
Ryan Poplin ◽  
Guillermo del Angel ◽  
...  
Keyword(s):  
Best Practices ◽  
Genome Analysis ◽  
High Confidence ◽  
Genome Analysis Toolkit

Big Data Analytics Framework for Real-Time Genome Analysis: A Comprehensive Approach

2019 ◽  
Vol 16 (8) ◽  
pp. 3419-3427
Author(s):  
Shishir K. Shandilya ◽  
S. Sountharrajan ◽  
Smita Shandilya ◽  
E. Suganya
Keyword(s):  
Big Data ◽  
Real Time ◽  
Genome Analysis ◽  
High Speed ◽  
High Performance ◽  
Big Data Analytics ◽  
Good Precision ◽  
Entire Genome ◽  
Big Data Technologies ◽  
Sequencing Platforms

Big Data Technologies are well-accepted in the recent years in bio-medical and genome informatics. They are capable to process gigantic and heterogeneous genome information with good precision and recall. With the quick advancements in computation and storage technologies, the cost of acquiring and processing the genomic data has decreased significantly. The upcoming sequencing platforms will produce vast amount of data, which will imperatively require high-performance systems for on-demand analysis with time-bound efficiency. Recent bio-informatics tools are capable of utilizing the novel features of Hadoop in a more flexible way. In particular, big data technologies such as MapReduce and Hive are able to provide high-speed computational environment for the analysis of petabyte scale datasets. This has attracted the focus of bio-scientists to use the big data applications to automate the entire genome analysis. The proposed framework is designed over MapReduce and Java on extended Hadoop platform to achieve the parallelism of Big Data Analysis. It will assist the bioinformatics community by providing a comprehensive solution for Descriptive, Comparative, Exploratory, Inferential, Predictive and Causal Analysis on Genome data. The proposed framework is user-friendly, fully-customizable, scalable and fit for comprehensive real-time genome analysis from data acquisition till predictive sequence analysis.

Semi-automated cancer genome analysis using high-performance computing

2017 ◽  
Vol 38 (10) ◽  
pp. 1325-1335 ◽  
Author(s):  
Giuliano Crispatzu ◽  
Pranav Kulkarni ◽  
Mohammad R. Toliat ◽  
Peter Nürnberg ◽  
Marco Herling ◽  
...  
Keyword(s):  
High Performance Computing ◽  
Genome Analysis ◽  
High Performance ◽  
Cancer Genome ◽  
Performance Computing

scholarly journals Sentieon DNA pipeline for variant detection - Software-only solution, over 20× faster than GATK 3.3 with identical results

2016 ◽  
Author(s):  
Jessica A. Weber ◽  
Rafael Aldana ◽  
Brendan D. Gallagher ◽  
Jeremy S. Edwards
Keyword(s):  
Dna Sequencing ◽  
Genome Analysis ◽  
Processing Speed ◽  
Best Practice ◽  
Secondary Analysis ◽  
Benchmark Analysis ◽  
Genome Analysis Toolkit ◽  
Variant Detection ◽  
Detection Software

Sentieon DNA Software is a suite of tools that allow running DNA sequencing secondary analysis pipelines. The Sentieon DNA Software produces results identical to the Genome Analysis Toolkit (GATK) Best Practice Workflow using HaplotypeCaller, with more than 20x increase in processing speed on the same hardware. This paper presents a benchmark analysis of both speed comparison and output concordance between using GATK and Sentieon DNA software on publically available datasets from the 100 genomes database.

scholarly journals The Genome Analysis Toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data

2010 ◽  
Vol 20 (9) ◽  
pp. 1297-1303 ◽  
Author(s):  
A. McKenna ◽  
M. Hanna ◽  
E. Banks ◽  
A. Sivachenko ◽  
K. Cibulskis ◽  
...  
Keyword(s):  
Dna Sequencing ◽  
Genome Analysis ◽  
Next Generation ◽  
Sequencing Data ◽  
Mapreduce Framework ◽  
Next Generation Dna Sequencing ◽  
Genome Analysis Toolkit

MitoFlex: an efficient, high-performance toolkit for animal mitogenome assembly, annotation, and visualization

2021 ◽  
Author(s):  
Jun-Yu Li ◽  
Wei-Xuan Li ◽  
An-Tai Wang ◽  
Zhang Yu
Keyword(s):  
Mitochondrial Genome ◽  
High Performance ◽  
High Throughput Sequencing ◽  
De Novo ◽  
Supplementary Information ◽  
Sequencing Data ◽  
Protein Coding ◽  
High Throughput Sequencing Data ◽  
Genome Analysis Toolkit ◽  
Overall Performance

Abstract Summary MitoFlex is a linux-based mitochondrial genome analysis toolkit, which provides a complete workflow of raw data filtering, de novo assembly, mitochondrial genome identification and annotation for animal high throughput sequencing data. The overall performance was compared between MitoFlex and its analogue MitoZ, in terms of protein coding gene recovery, memory consumption and processing speed. Availability MitoFlex is available at https://github.com/Prunoideae/MitoFlex under GPLv3 license. Supplementary information Supplementary data are available at Bioinformatics online.

Sign in / Sign up

Export Citation Format

Share Document