scholarly journals VC@Scale: Scalable and high-performance variant calling on cluster environments

GigaScience ◽  
2021 ◽  
Vol 10 (9) ◽  
Author(s):  
Tanveer Ahmad ◽  
Zaid Al Ars ◽  
H Peter Hofstee
Keyword(s):  
Deep Learning ◽  
High Performance ◽  
High Efficiency ◽  
Variant Calling ◽  
Apache Spark ◽  
Next Generation Sequencing Data ◽  
Sequencing Data ◽  
Single Node ◽  
Loosely Coupled ◽  
Gpu Clusters

Abstract Background Recently many new deep learning–based variant-calling methods like DeepVariant have emerged as more accurate compared with conventional variant-calling algorithms such as GATK HaplotypeCaller, Sterlka2, and Freebayes albeit at higher computational costs. Therefore, there is a need for more scalable and higher performance workflows of these deep learning methods. Almost all existing cluster-scaled variant-calling workflows that use Apache Spark/Hadoop as big data frameworks loosely integrate existing single-node pre-processing and variant-calling applications. Using Apache Spark just for distributing/scheduling data among loosely coupled applications or using I/O-based storage for storing the output of intermediate applications does not exploit the full benefit of Apache Spark in-memory processing. To achieve this, we propose a native Spark-based workflow that uses Python and Apache Arrow to enable efficient transfer of data between different workflow stages. This benefits from the ease of programmability of Python and the high efficiency of Arrow’s columnar in-memory data transformations. Results Here we present a scalable, parallel, and efficient implementation of next-generation sequencing data pre-processing and variant-calling workflows. Our design tightly integrates most pre-processing workflow stages, using Spark built-in functions to sort reads by coordinates and mark duplicates efficiently. Our approach outperforms state-of-the-art implementations by >2 times for the pre-processing stages, creating a scalable and high-performance solution for DeepVariant for both CPU-only and CPU + GPU clusters. Conclusions We show the feasibility and easy scalability of our approach to achieve high performance and efficient resource utilization for variant-calling analysis on high-performance computing clusters using the standardized Apache Arrow data representations. All codes, scripts, and configurations used to run our implementations are publicly available and open sourced; see https://github.com/abs-tudelft/variant-calling-at-scale.

scholarly journals SLR-superscaffolder: a de novo scaffolding tool for synthetic long reads using a top-to-bottom scheme

2021 ◽  
Vol 22 (1) ◽  
Author(s):  
Lidong Guo ◽  
Mengyang Xu ◽  
Wenchao Wang ◽  
Shengqiang Gu ◽  
Xia Zhao ◽  
...  
Keyword(s):  
High Efficiency ◽  
De Novo ◽  
Next Generation Sequencing Data ◽  
Sequencing Data ◽  
Draft Assembly ◽  
Screening Algorithm ◽  
Long Reads ◽  
Hybrid Genome ◽  
Genomics Research ◽  
Negative Effect

Abstract Background Synthetic long reads (SLR) with long-range co-barcoding information are now widely applied in genomics research. Although several tools have been developed for each specific SLR technique, a robust standalone scaffolder with high efficiency is warranted for hybrid genome assembly. Results In this work, we developed a standalone scaffolding tool, SLR-superscaffolder, to link together contigs in draft assemblies using co-barcoding and paired-end read information. Our top-to-bottom scheme first builds a global scaffold graph based on Jaccard Similarity to determine the order and orientation of contigs, and then locally improves the scaffolds with the aid of paired-end information. We also exploited a screening algorithm to reduce the negative effect of misassembled contigs in the input assembly. We applied SLR-superscaffolder to a human single tube long fragment read sequencing dataset and increased the scaffold NG50 of its corresponding draft assembly 1349 fold. Moreover, benchmarking on different input contigs showed that this approach overall outperformed existing SLR scaffolders, providing longer contiguity and fewer misassemblies, especially for short contigs assembled by next-generation sequencing data. The open-source code of SLR-superscaffolder is available at https://github.com/BGI-Qingdao/SLR-superscaffolder. Conclusions SLR-superscaffolder can dramatically improve the contiguity of a draft assembly by integrating a hybrid assembly strategy.

scholarly journals SparkRA: Enabling Big Data Scalability for the GATK RNA-seq Pipeline with Apache Spark

Genes ◽  
2020 ◽  
Vol 11 (1) ◽  
pp. 53
Author(s):  
Zaid Al-Ars ◽  
Saiyi Wang ◽  
Hamid Mushtaq
Keyword(s):  
Low Cost ◽  
Computing Time ◽  
Scale Up ◽  
Variant Calling ◽  
Computation Time ◽  
Apache Spark ◽  
Rna Seq ◽  
Single Node ◽  
Practical Applications ◽  
Computational Resources

The rapid proliferation of low-cost RNA-seq data has resulted in a growing interest in RNA analysis techniques for various applications, ranging from identifying genotype–phenotype relationships to validating discoveries of other analysis results. However, many practical applications in this field are limited by the available computational resources and associated long computing time needed to perform the analysis. GATK has a popular best practices pipeline specifically designed for variant calling RNA-seq analysis. Some tools in this pipeline are not optimized to scale the analysis to multiple processors or compute nodes efficiently, thereby limiting their ability to process large datasets. In this paper, we present SparkRA, an Apache Spark based pipeline to efficiently scale up the GATK RNA-seq variant calling pipeline on multiple cores in one node or in a large cluster. On a single node with 20 hyper-threaded cores, the original pipeline runs for more than 5 h to process a dataset of 32 GB. In contrast, SparkRA is able to reduce the overall computation time of the pipeline on the same single node by about 4×, reducing the computation time down to 1.3 h. On a cluster with 16 nodes (each with eight single-threaded cores), SparkRA is able to further reduce this computation time by 7.7× compared to a single node. Compared to other scalable state-of-the-art solutions, SparkRA is 1.2× faster while achieving the same accuracy of the results.

scholarly journals PathoQC: Computationally Efficient Read Preprocessing and Quality Control for High-Throughput Sequencing Data Sets

2014 ◽  
Vol 13s1 ◽  
pp. CIN.S13890 ◽  
Author(s):  
Changjin Hong ◽  
Solaiappan Manimaran ◽  
William Evan Johnson
Keyword(s):  
Quality Control ◽  
High Throughput ◽  
High Performance ◽  
High Throughput Sequencing ◽  
Next Generation Sequencing Data ◽  
Data Sets ◽  
Sequencing Data ◽  
Computationally Efficient ◽  
High Throughput Sequencing Data ◽  
Downstream Analysis

Quality control and read preprocessing are critical steps in the analysis of data sets generated from high-throughput genomic screens. In the most extreme cases, improper preprocessing can negatively affect downstream analyses and may lead to incorrect biological conclusions. Here, we present PathoQC, a streamlined toolkit that seamlessly combines the benefits of several popular quality control software approaches for preprocessing next-generation sequencing data. PathoQC provides a variety of quality control options appropriate for most high-throughput sequencing applications. PathoQC is primarily developed as a module in the PathoScope software suite for metagenomic analysis. However, PathoQC is also available as an open-source Python module that can run as a stand-alone application or can be easily integrated into any bioinformatics workflow. PathoQC achieves high performance by supporting parallel computation and is an effective tool that removes technical sequencing artifacts and facilitates robust downstream analysis. The PathoQC software package is available at http://sourceforge.net/projects/PathoScope/ .

scholarly journals Lacer: accurate base quality score recalibration for improving variant calling from next-generation sequencing data in any organism

2017 ◽  
Author(s):  
Jade C.S. Chung ◽  
Swaine L. Chen
Keyword(s):  
Next Generation Sequencing ◽  
Variant Calling ◽  
Quality Score ◽  
Identification Accuracy ◽  
Next Generation Sequencing Data ◽  
Sequencing Error ◽  
Next Generation ◽  
Sequencing Data ◽  
Base Quality Score ◽  
Generation Sequencing

AbstractNext-generation sequencing data is accompanied by quality scores that quantify sequencing error. Inaccuracies in these quality scores propagate through all subsequent analyses; thus base quality score recalibration is a standard step in many next-generation sequencing workflows, resulting in improved variant calls. Current base quality score recalibration algorithms rely on the assumption that sequencing errors are already known; for human resequencing data, relatively complete variant databases facilitate this. However, because existing databases are still incomplete, recalibration is still inaccurate; and most organisms do not have variant databases, exacerbating inaccuracy for non-human data. To overcome these logical and practical problems, we introduce Lacer, which recalibrates base quality scores without assuming knowledge of correct and incorrect bases and without requiring knowledge of common variants. Lacer is the first logically sound, fully general, and truly accurate base recalibrator. Lacer enhances variant identification accuracy for resequencing data of human as well as other organisms (which are not accessible to current recalibrators), simultaneously improving and extending the benefits of base quality score recalibration to nearly all ongoing sequencing projects. Lacer is available at: https://github.com/swainechen/lacer.

scholarly journals DeNovoCNN: A deep learning approach to de novo variant calling in next generation sequencing data

2021 ◽  
Author(s):  
Gelana Khazeeva ◽  
Karolis Sablauskas ◽  
Bart van der Sanden ◽  
Wouter Steyaert ◽  
Michael Kwint ◽  
...  
Keyword(s):  
Exome Sequencing ◽  
De Novo ◽  
Genetic Disorders ◽  
Variant Calling ◽  
Next Generation Sequencing Data ◽  
Sequencing Data ◽  
Accurate Identification ◽  
Whole Exome ◽  
De Novo Variant ◽  
Generation Sequencing

De novo mutations (DNMs) are an important cause of genetic disorders. The accurate identification of DNMs from sequencing data is therefore fundamental to rare disease research and diagnostics. Unfortunately, identifying reliable DNMs remains a major challenge due to sequence errors, uneven coverage, and mapping artifacts. Here, we developed a deep convolutional neural network (CNN) DNM caller (DeNovoCNN), that encodes alignment of sequence reads for a trio as 160×164 resolution images. DeNovoCNN was trained on DNMs of whole exome sequencing (WES) of 2003 trios achieving on average 99.2% recall and 93.8% precision. We find that DeNovoCNN has increased recall/sensitivity and precision compared to existing de novo calling approaches (GATK, DeNovoGear, Samtools) based on the Genome in a Bottle reference dataset. Sanger validations of DNMs called in both exome and genome datasets confirm that DeNovoCNN outperforms existing methods. Most importantly, we show that DeNovoCNN is robust against different exome sequencing and analyses approaches, thereby allowing it to be applied on other datasets. DeNovoCNN is freely available and can be run on existing alignment (BAM/CRAM) and variant calling (VCF) files from WES and WGS without a need for variant recalling.

scholarly journals Pisces: An Accurate and Versatile Variant Caller for Somatic and Germline Next-Generation Sequencing Data

2018 ◽  
Author(s):  
Tamsen Dunn ◽  
Gwenn Berry ◽  
Dorothea Emig-Agius ◽  
Yu Jiang ◽  
Serena Lei ◽  
...  
Keyword(s):  
Next Generation Sequencing ◽  
Gene Mutations ◽  
Variant Calling ◽  
Amplicon Sequencing ◽  
Supplementary Information ◽  
Next Generation Sequencing Data ◽  
Next Generation ◽  
Sequencing Data ◽  
Ras Gene ◽  
Generation Sequencing

AbstractMotivationNext-Generation Sequencing (NGS) technology is transitioning quickly from research labs to clinical settings. The diagnosis and treatment selection for many acquired and autosomal conditions necessitate a method for accurately detecting somatic and germline variants, suitable for the clinic.ResultsWe have developed Pisces, a rapid, versatile and accurate small variant calling suite designed for somatic and germline amplicon sequencing applications. Pisces accuracy is achieved by four distinct modules, the Pisces Read Stitcher, Pisces Variant Caller, the Pisces Variant Quality Recalibrator, and the Pisces Variant Phaser. Each module incorporates a number of novel algorithmic strategies aimed at reducing noise or increasing the likelihood of detecting a true variant.AvailabilityPisces is distributed under an open source license and can be downloaded from https://github.com/Illumina/Pisces. Pisces is available on the BaseSpace™ SequenceHub as part of the TruSeq Amplicon workflow and the Illumina Ampliseq Workflow. Pisces is distributed on Illumina sequencing platforms such as the MiSeq™, and is included in the Praxis™ Extended RAS Panel test which was recently approved by the FDA for the detection of multiple RAS gene [email protected] informationSupplementary data are available online.

scholarly journals NGSphy: phylogenomic simulation of next-generation sequencing data

2017 ◽  
Author(s):  
Merly Escalona ◽  
Sara Rocha ◽  
David Posada
Keyword(s):  
Next Generation Sequencing ◽  
Variant Calling ◽  
Gene Families ◽  
Common Species ◽  
Next Generation Sequencing Data ◽  
Phylogenomic Analysis ◽  
Next Generation ◽  
Sequencing Data ◽  
Sequencing Technologies ◽  
Generation Sequencing

AbstractMotivationAdvances in sequencing technologies have made it feasible to obtain massive datasets for phylogenomic inference, often consisting of large numbers of loci from multiple species and individuals. The phylogenomic analysis of next-generation sequencing (NGS) data implies a complex computational pipeline where multiple technical and methodological decisions are necessary that can influence the final tree obtained, like those related to coverage, assembly, mapping, variant calling and/or phasing.ResultsTo assess the influence of these variables we introduce NGSphy, an open-source tool for the simulation of Illumina reads/read counts obtained from haploid/diploid individual genomes with thousands of independent gene families evolving under a common species tree. In order to resemble real NGS experiments, NGSphy includes multiple options to model sequencing coverage (depth) heterogeneity across species, individuals and loci, including off-target or uncaptured loci. For comprehensive simulations covering multiple evolutionary scenarios, parameter values for the different replicates can be sampled from user-defined statistical distributions.AvailabilitySource code, full documentation and tutorials including a quick start guide are available at http://github.com/merlyescalona/[email protected]. [email protected]

scholarly journals Disambiguate: An open-source application for disambiguating two species in next generation sequencing data from grafted samples

F1000Research ◽  
2016 ◽  
Vol 5 ◽  
pp. 2741 ◽  
Author(s):  
Miika J. Ahdesmäki ◽  
Simon R. Gray ◽  
Justin H. Johnson ◽  
Zhongwu Lai
Keyword(s):  
Open Source ◽  
Variant Calling ◽  
High Sensitivity ◽  
Next Generation Sequencing Data ◽  
Sequencing Data ◽  
Novel Approach ◽  
Gene Expression Quantification ◽  
Closed Source ◽  
Generation Sequencing ◽  
Bioinformatics Community

Grafting of cell lines and primary tumours is a crucial step in the drug development process between cell line studies and clinical trials. Disambiguate is a program for computationally separating the sequencing reads of two species derived from grafted samples. Disambiguate operates on alignments to the two species and separates the components at very high sensitivity and specificity as illustrated in artificially mixed human-mouse samples. This allows for maximum recovery of data from target tumours for more accurate variant calling and gene expression quantification. Given that no general use open source algorithm accessible to the bioinformatics community exists for the purposes of separating the two species data, the proposed Disambiguate tool presents a novel approach and improvement to performing sequence analysis of grafted samples. Both Python and C++ implementations are available and they are integrated into several open and closed source pipelines. Disambiguate is open source and is freely available at https://github.com/AstraZeneca-NGS/disambiguate.

Sign in / Sign up

Export Citation Format

Share Document