scholarly journals BdBG: a bucket-based method for compressing genome sequencing data with dynamic de Bruijn graphs

PeerJ ◽  
2018 ◽  
Vol 6 ◽  
pp. e5611 ◽  
Author(s):  
Rongjie Wang ◽  
Junyi Li ◽  
Yang Bai ◽  
Tianyi Zang ◽  
Yadong Wang
Keyword(s):  
Data Compression ◽  
Genome Sequencing ◽  
De Bruijn Graph ◽  
Storage Space ◽  
Sequencing Data ◽  
De Bruijn Graphs ◽  
Efficient Data ◽  
De Bruijn ◽  
Next Generation Sequencing Ngs ◽  
Ngs Data

Dramatic increases in data produced by next-generation sequencing (NGS) technologies demand data compression tools for saving storage space. However, effective and efficient data compression for genome sequencing data has remained an unresolved challenge in NGS data studies. In this paper, we propose a novel alignment-free and reference-free compression method, BdBG, which is the first to compress genome sequencing data with dynamic de Bruijn graphs based on the data after bucketing. Compared with existing de Bruijn graph methods, BdBG only stored a list of bucket indexes and bifurcations for the raw read sequences, and this feature can effectively reduce storage space. Experimental results on several genome sequencing datasets show the effectiveness of BdBG over three state-of-the-art methods. BdBG is written in python and it is an open source software distributed under the MIT license, available for download at https://github.com/rongjiewang/BdBG.

scholarly journals Succinct Dynamic de Bruijn Graphs

2020 ◽  
Author(s):  
Bahar Alipanahi ◽  
Alan Kuhnle ◽  
Simon J. Puglisi ◽  
Leena Salmela ◽  
Christina Boucher
Keyword(s):  
Data Structures ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Efficient Manner ◽  
De Bruijn Graphs ◽  
High Throughput Sequencing Data ◽  
Efficient Data ◽  
De Bruijn

AbstractMotivationThe de Bruijn graph is one of the fundamental data structures for analysis of high throughput sequencing data. In order to be applicable to population-scale studies, it is essential to build and store the graph in a space- and time-efficient manner. In addition, due to the ever-changing nature of population studies, it has become essential to update the graph after construction e.g. add and remove nodes and edges. Although there has been substantial effort on making the construction and storage of the graph efficient, there is a limited amount of work in building the graph in an efficient and mutable manner. Hence, most space efficient data structures require complete reconstruction of the graph in order to add or remove edges or nodes.ResultsIn this paper we present DynamicBOSS, a succinct representation of the de Bruijn graph that allows for an unlimited number of additions and deletions of nodes and edges. We compare our method with other competing methods and demonstrate that DynamicBOSS is the only method that supports both addition and deletion and is applicable to very large samples (e.g. greater than 15 billion k-mers). Competing dynamic methods e.g., FDBG (Crawford et al., 2018) cannot be constructed on large scale datasets, or cannot support both addition and deletion e.g., BiFrost (Holley and Melsted, 2019).AvailabilityDynamicBOSS is publicly available at https://github.com/baharpan/[email protected]

scholarly journals Accurate determination of node and arc multiplicities in de bruijn graphs using conditional random fields

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Aranka Steyaert ◽  
Pieter Audenaert ◽  
Jan Fostier
Keyword(s):  
Genomic Sequence ◽  
Conditional Random Field ◽  
Accurate Determination ◽  
Next Generation Sequencing Data ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
De Bruijn Graphs ◽  
Sequencing Errors ◽  
Expectation Maximisation ◽  
De Bruijn

Abstract Background De Bruijn graphs are key data structures for the analysis of next-generation sequencing data. They efficiently represent the overlap between reads and hence, also the underlying genome sequence. However, sequencing errors and repeated subsequences render the identification of the true underlying sequence difficult. A key step in this process is the inference of the multiplicities of nodes and arcs in the graph. These multiplicities correspond to the number of times each k-mer (resp. k+1-mer) implied by a node (resp. arc) is present in the genomic sequence. Determining multiplicities thus reveals the repeat structure and presence of sequencing errors. Multiplicities of nodes/arcs in the de Bruijn graph are reflected in their coverage, however, coverage variability and coverage biases render their determination ambiguous. Current methods to determine node/arc multiplicities base their decisions solely on the information in nodes and arcs individually, under-utilising the information present in the sequencing data. Results To improve the accuracy with which node and arc multiplicities in a de Bruijn graph are inferred, we developed a conditional random field (CRF) model to efficiently combine the coverage information within each node/arc individually with the information of surrounding nodes and arcs. Multiplicities are thus collectively assigned in a more consistent manner. Conclusions We demonstrate that the CRF model yields significant improvements in accuracy and a more robust expectation-maximisation parameter estimation. True k-mers can be distinguished from erroneous k-mers with a higher F1 score than existing methods. A C++11 implementation is available at https://github.com/biointec/detoxunder the GNU AGPL v3.0 license.

scholarly journals Buffering Updates Enables Efficient Dynamic de Bruijn Graphs

2021 ◽  
Author(s):  
Jarno Alanko ◽  
Bahar Alipanahi ◽  
Jonathen Settle ◽  
Christina Boucher ◽  
Travis Gagie
Keyword(s):  
Graph Model ◽  
Biological Data ◽  
Theory And Practice ◽  
De Bruijn Graph ◽  
Efficient Manner ◽  
De Bruijn Graphs ◽  
Trade Offs ◽  
Order Of Magnitude ◽  
Efficient Data ◽  
De Bruijn

AbstractMotivationThe de Bruijn graph has become a ubiquitous graph model for biological data ever since its initial introduction in the later 1990s. It has been used for a variety of purposes including genome assembly (Zerbino and Birney, 2008; Bankevich et al., 2012b; Peng et al., 2012), variant detection (Alipanahi et al., 2020b; Iqbal et al., 2012), and storage of assembled genomes (Chikhi et al., 2016). For this reason, there have been over a dozen methods for building and representing the de Bruijn graph and its variants in a space- and time-efficient manner.ResultsAlthough there exists a plethora of space efficient data structures for storing the de Bruijn graph, the majority of them make a compression-mutability trade-off. In particular, with the exception of a few methods (Muggli et al., 2019; Holley and Melsted, 2020; Crawford et al., 2018), compressed and compact de Bruijn graphs do not allow for the graph to be efficiently updated, allowing for data to be added or deleted. The most recent compressed dynamic de Bruijn graph, (Alipanahi et al., 2020a), relies on dynamic bit vectors, which are slow in theory and practice. To address this shortcoming, we present BufBOSS which is a compressed dynamic de Bruijn graph that removes the necessity of dynamic bit vectors by buffering data that should be added or removed from the graph. We implement our method, which we refer to as BufBOSS, and compare its performance to Bifrost, DynamicBOSS, and FDBG. Our experiments demonstrate that BufBOSS achieves attractive trade-offs compared to other tools in terms of time, memory and disk, and has the best deletion performance by an order of magnitude out of all the tools that are able to perform deletions. Our implementation is available at https://github.com/jnalanko/[email protected]

Inference of viral quasispecies with a paired de Bruijn graph

2020 ◽  
Author(s):  
Borja Freire ◽  
Susana Ladra ◽  
Jose R Paramá ◽  
Leena Salmela
Keyword(s):  
High Throughput Sequencing ◽  
De Novo ◽  
Supplementary Information ◽  
De Bruijn Graph ◽  
Viral Quasispecies ◽  
Sequencing Data ◽  
De Bruijn Graphs ◽  
Sequencing Errors ◽  
High Throughput Sequencing Data ◽  
De Bruijn

Abstract Motivation RNA viruses exhibit a high mutation rate and thus they exist in infected cells as a population of closely related strains called viral quasispecies. The viral quasispecies assembly problem asks to characterize the quasispecies present in a sample from high-throughput sequencing data. We study the de novo version of the problem, where reference sequences of the quasispecies are not available. Current methods for assembling viral quasispecies are either based on overlap graphs or on de Bruijn graphs. Overlap graph-based methods tend to be accurate but slow, whereas de Bruijn graph-based methods are fast but less accurate. Results We present viaDBG, which is a fast and accurate de Bruijn graph-based tool for de novo assembly of viral quasispecies. We first iteratively correct sequencing errors in the reads, which allows us to use large k-mers in the de Bruijn graph. To incorporate the paired-end information in the graph, we also adapt the paired de Bruijn graph for viral quasispecies assembly. These features enable the use of long-range information in contig construction without compromising the speed of de Bruijn graph-based approaches. Our experimental results show that viaDBG is both accurate and fast, whereas previous methods are either fast or accurate but not both. In particular, viaDBG has comparable or better accuracy than SAVAGE, while being at least nine times faster. Furthermore, the speed of viaDBG is comparable to PEHaplo but viaDBG is able to retrieve also low abundance quasispecies, which are often missed by PEHaplo. Availability and implementation viaDBG is implemented in C++ and it is publicly available at https://bitbucket.org/bfreirec1/viadbg. All datasets used in this article are publicly available at https://bitbucket.org/bfreirec1/data-viadbg/. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Bidirectional Variable-Order de Bruijn Graphs

2018 ◽  
Vol 29 (08) ◽  
pp. 1279-1295 ◽  
Author(s):  
Djamal Belazzougui ◽  
Travis Gagie ◽  
Veli Mäkinen ◽  
Marco Previtali ◽  
Simon J. Puglisi
Keyword(s):  
Data Compression ◽  
Graph Representation ◽  
De Bruijn Graph ◽  
Variable Order ◽  
Suffix Trees ◽  
De Bruijn Graphs ◽  
Small Constant ◽  
Left And Right ◽  
De Bruijn ◽  
The Cost

Compressed suffix trees and bidirectional FM-indexes can store a set of strings and support queries that let us explore the set of substrings they contain, adding and deleting characters on both the left and right, but they can use much more space than a de Bruijn graph for the strings. Bowe et al.’s BWT-based de Bruijn graph representation (Proc. Workshop on Algorithms for Bioinformatics, pp. 225–235, 2012) can be made bidirectional as well, at the cost of increasing its space usage by a small constant, but it fixes the length of the substrings. Boucher et al. (Proc. Data Compression Conference, pp. 383–392, 2015) generalized Bowe et al.’s representation to support queries about variable-length substrings, but at the cost of bidirectionality. In this paper we show how to make Boucher et al.’s variable-order implementation of de Bruijn graphs bidirectional.

scholarly journals Integrating long-range connectivity information into de Bruijn graphs

2017 ◽  
Author(s):  
Isaac Turner ◽  
Kiran V Garimella ◽  
Zamin Iqbal ◽  
Gil McVean
Keyword(s):  
Data Structure ◽  
Long Range ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Genomic Context ◽  
De Bruijn Graphs ◽  
Connectivity Information ◽  
String Graph ◽  
Free Data ◽  
De Bruijn

AbstractMotivationThe de Bruijn graph is a simple and efficient data structure that is used in many areas of sequence analysis including genome assembly, read error correction and variant calling. The data structure has a single parameter k, is straightforward to implement and is tractable for large genomes with high sequencing depth. It also enables representation of multiple samples simultaneously to facilitate comparison. However, unlike the string graph, a de Bruijn graph does not retain long range information that is inherent in the read data. For this reason, applications that rely on de Bruijn graphs can produce sub-optimal results given their input.ResultsWe present a novel assembly graph data structure: the Linked de Bruijn Graph (LdBG). Constructed by adding annotations on top of a de Bruijn graph, it stores long range connectivity information through the graph. We show that with error-free data it is possible to losslessly store and recover sequence from a Linked de Bruijn graph. With assembly simulations we demonstrate that the LdBG data structure outperforms both the de Bruijn graph and the String Graph Assembler (SGA). Finally we apply the LdBG to Klebsiella pneumoniae short read data to make large (12 kbp) variant calls, which we validate using PacBio sequencing data, and to characterise the genomic context of drug-resistance genes.AvailabilityLinked de Bruijn Graphs and associated algorithms are implemented as part of McCortex, available under the MIT license at https://github.com/mcvean/[email protected].

scholarly journals Succinct Dynamic de Bruijn Graphs

2020 ◽  
Author(s):  
Bahar Alipanahi ◽  
Alan Kuhnle ◽  
Simon J Puglisi ◽  
Leena Salmela ◽  
Christina Boucher
Keyword(s):  
Data Structures ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
Supplementary Information ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Efficient Manner ◽  
De Bruijn Graphs ◽  
High Throughput Sequencing Data ◽  
De Bruijn

Abstract Motivation The de Bruijn graph is one of the fundamental data structures for analysis of high throughput sequencing data. In order to be applicable to population-scale studies, it is essential to build and store the graph in a space- and time- efficient manner. In addition, due to the ever-changing nature of population studies, it has become essential to update the graph after construction e.g. add and remove nodes and edges. Although there has been substantial effort on making the construction and storage of the graph efficient, there is a limited amount of work in building the graph in an efficient and mutable manner. Hence, most space efficient data structures require complete reconstruction of the graph in order to add or remove edges or nodes. Results In this paper we present DynamicBOSS, a succinct representation of the de Bruijn graph that allows for an unlimited number of additions and deletions of nodes and edges. We compare our method with other competing methods and demonstrate that DynamicBOSS is the only method that supports both addition and deletion and is applicable to very large samples (e.g. greater than 15 billion k-mers). Competing dynamic methods e.g., FDBG (Crawford et al., 2018) cannot be constructed on large scale datasets, or cannot support both addition and deletion e.g., BiFrost (Holley and Melsted, 2019). Availability DynamicBOSS is publicly available at https://github.com/baharpan/dynboss. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals LOGAN: A framework for LOssless Graph-based ANalysis of high throughput sequence data

2017 ◽  
Author(s):  
Anthony Bolger ◽  
Alisandra Denton ◽  
Marie Bolger ◽  
Björn Usadel
Keyword(s):  
Data Structure ◽  
Data Structures ◽  
Sequence Data ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Solanum Pennellii ◽  
Desktop Computer ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
Memory Efficient

AbstractRecent massive growth in the production of sequencing data necessitates matching improve-ments in bioinformatics tools to effectively utilize it. Existing tools suffer from limitations in both scalability and applicability which are inherent to their underlying algorithms and data structures. We identify the key requirements for the ideal data structure for sequence analy-ses: it should be informationally lossless, locally updatable, and memory efficient; requirements which are not met by data structures underlying the major assembly strategies Overlap Layout Consensus and De Bruijn Graphs. We therefore propose a new data structure, the LOGAN graph, which is based on a memory efficient Sparse De Bruijn Graph with routing information. Innovations in storing routing information and careful implementation allow sequence datasets for Escherichia coli (4.6Mbp, 117x coverage), Arabidopsis thaliana (135Mbp, 17.5x coverage) and Solanum pennellii (1.2Gbp, 47x coverage) to be loaded into memory on a desktop computer in seconds, minutes, and hours respectively. Memory consumption is competitive with state of the art alternatives, while losslessly representing the reads in an indexed and updatable form. Both Second and Third Generation Sequencing reads are supported. Thus, the LOGAN graph is positioned to be the backbone for major breakthroughs in sequence analysis such as integrated hybrid assembly, assembly of exceptionally large and repetitive genomes, as well as assembly and representation of pan-genomes.

scholarly journals Reference-free comparison of microbial communities via de Bruijn graphs

2016 ◽  
Author(s):  
Serghei Mangul ◽  
David Koslicki
Keyword(s):  
Microbial Communities ◽  
High Throughput Sequencing ◽  
Microbial Community Composition ◽  
De Bruijn Graph ◽  
Data Sets ◽  
Sequencing Data ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
The Individual ◽  
And Control

ABSTRACTMicrobial communities inhabiting the human body exhibit significant variability across different individuals and tissues, and are suggested to play an important role in health and disease. High-throughput sequencing offers unprecedented possibilities to profile microbial community composition, but limitations of existing taxonomic classification methods (including incompleteness of existing microbial reference databases) limits the ability to accurately compare microbial communities across different samples. In this paper, we present a method able to overcome these limitations by circumventing the classification step and directly using the sequencing data to compare microbial communities. The proposed method provides a powerful reference-free way to assess differences in microbial abundances across samples. This method, called EMDeBruijn, condenses the sequencing data into a de Bruijn graph. The Earth Mover's Distance (EMD) is then used to measure similarities and differences of the microbial communities associated with the individual graphs. We apply this method to RNA-Seq data sets from a coronary artery calcification (CAC) study and shown that EMDeBruijn is able to differentiate between case and control CAC samples while utilizing all the candidate microbial reads. We compare these results to current reference-based methods, which are shown to have a limited capacity to discriminate between case and control samples. We conclude that this reference-free approach is a viable choice in comparative metatranscriptomic studies.

scholarly journals Scalable, ultra-fast, and low-memory construction of compacted de Bruijn graphs with Cuttlefish 2

2021 ◽  
Author(s):  
Jamshed Khan ◽  
Marek Kokot ◽  
Sebastian Deorowicz ◽  
Rob Patro
Keyword(s):  
Open Source Software ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Bacterial Genomes ◽  
De Bruijn Graphs ◽  
Genomic Analyses ◽  
Efficient Construction ◽  
De Bruijn ◽  
Computational Bottleneck ◽  
Reference Genomes

The de Bruijn graph has become a key data structure in modern computational genomics, and of keen interest is its compacted variant. The compacted de Bruijn graph provides a lossless representation of the graph, and it is often considerably more efficient to store and process than its non-compacted counterpart. Construction of the compacted de Bruijn graph resides upstream of many genomic analyses. As the quantity of sequencing data and the number of reference genomes on which to perform these analyses grow rapidly, efficient construction of the compacted graph becomes a computational bottleneck for these tasks. We present Cuttlefish 2, significantly advancing the existing state-of-the-art methods for construction of this graph. On a typical shared-memory machine, it reduces the construction of the compacted de Bruijn graph for 661K bacterial genomes (2.58 Tbp of input reference genomes) from about 4.5 days to 17—23 hours. Similarly on sequencing data, it constructs the graph for a 1.52 Tbp white spruce read set in about 10 hours, while the closest competitor, which also uses considerably more memory, requires 54—58 hours. Cuttlefish 2 is implemented in C++14, and is available as open-source software under a BSD-3-Clause license at https://github.com/COMBINE-lab/cuttlefish.

Sign in / Sign up

Export Citation Format

Share Document