scholarly journals Genozip - A Universal Extensible Genomic Data Compressor

2021 ◽  
Author(s):  
Divon Lan ◽  
Ray Tobler ◽  
Yassine Souilmi ◽  
Bastien Llamas
Keyword(s):  
High Performance ◽  
Genomic Data ◽  
General Purpose ◽  
Supplementary Information ◽  
Data Types ◽  
Data Formats ◽  
Development Framework ◽  
File Formats ◽  
Genomics Research ◽  
Core Capabilities

Abstract We present Genozip, a universal and fully featured compression software for genomic data. Genozip is designed to be a general-purpose software and a development framework for genomic compression by providing five core capabilities – universality (support for all common genomic file formats), high compression ratios, speed, feature-richness, and extensibility. Genozip delivers high-performance compression for widely-used genomic data formats in genomics research, namely FASTQ, SAM/BAM/CRAM, VCF, GVF, FASTA, PHYLIP, and 23andMe formats. Our test results show that Genozip is fast and achieves greatly improved compression ratios, even when the files are already compressed. Further, Genozip is architected with a separation of the Genozip Framework from file-format-specific Segmenters and data-type-specific Codecs. With this, we intend for Genozip to be a general-purpose compression platform where researchers can implement compression for additional file formats, as well as new codecs for data types or fields within files, in the future. We anticipate that this will ultimately increase the visibility and adoption of these algorithms by the user community, thereby accelerating further innovation in this space. Availability: Genozip is written in C. The code is open-source and available on GitHub (https://github.com/divonlan/genozip). The package is free for non-commercial use. It is distributed as a Docker container on DockerHub and through the conda package manager. Genozip is tested on Linux, Mac, and Windows. Supplementary information: Supplementary data are available at Bioinformatics online.

scholarly journals Augmented Interval List: a novel data structure for efficient genomic interval search

2019 ◽  
Vol 35 (23) ◽  
pp. 4907-4911 ◽  
Author(s):  
Jianglin Feng ◽  
Aakrosh Ratan ◽  
Nathan C Sheffield
Keyword(s):  
Data Structure ◽  
High Performance ◽  
Genomic Analysis ◽  
Genomic Data ◽  
Interval Data ◽  
Supplementary Information ◽  
Genomic Interval ◽  
Interval Trees ◽  
Running Maximum ◽  
Scalable Methods

Abstract Motivation Genomic data is frequently stored as segments or intervals. Because this data type is so common, interval-based comparisons are fundamental to genomic analysis. As the volume of available genomic data grows, developing efficient and scalable methods for searching interval data is necessary. Results We present a new data structure, the Augmented Interval List (AIList), to enumerate intersections between a query interval q and an interval set R. An AIList is constructed by first sorting R as a list by the interval start coordinate, then decomposing it into a few approximately flattened components (sublists), and then augmenting each sublist with the running maximum interval end. The query time for AIList is O(log2N+n+m), where n is the number of overlaps between R and q, N is the number of intervals in the set R and m is the average number of extra comparisons required to find the n overlaps. Tested on real genomic interval datasets, AIList code runs 5–18 times faster than standard high-performance code based on augmented interval-trees, nested containment lists or R-trees (BEDTools). For large datasets, the memory-usage for AIList is 4–60% of other methods. The AIList data structure, therefore, provides a significantly improved fundamental operation for highly scalable genomic data analysis. Availability and implementation An implementation of the AIList data structure with both construction and search algorithms is available at http://ailist.databio.org. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Accelerated FDPS: Algorithms to use accelerators with FDPS

2020 ◽  
Vol 72 (1) ◽  
Author(s):  
Masaki Iwasawa ◽  
Daisuke Namekata ◽  
Keigo Nitadori ◽  
Kentaro Nomura ◽  
Long Wang ◽  
...  
Keyword(s):  
Graphics Processing Units ◽  
High Performance ◽  
General Purpose ◽  
Performance Model ◽  
Performance Tuning ◽  
Data Types ◽  
Interaction Function ◽  
Current Implementation ◽  
And Performance ◽  
Graphics Processing

Abstract We describe algorithms implemented in FDPS (Framework for Developing Particle Simulators) to make efficient use of accelerator hardware such as GPGPUs (general-purpose computing on graphics processing units). We have developed FDPS to make it possible for researchers to develop their own high-performance parallel particle-based simulation programs without spending large amounts of time on parallelization and performance tuning. FDPS provides a high-performance implementation of parallel algorithms for particle-based simulations in a “generic” form, so that researchers can define their own particle data structure and interparticle interaction functions. FDPS compiled with user-supplied data types and interaction functions provides all the necessary functions for parallelization, and researchers can thus write their programs as though they are writing simple non-parallel code. It has previously been possible to use accelerators with FDPS by writing an interaction function that uses the accelerator. However, the efficiency was limited by the latency and bandwidth of communication between the CPU and the accelerator, and also by the mismatch between the available degree of parallelism of the interaction function and that of the hardware parallelism. We have modified the interface of the user-provided interaction functions so that accelerators are more efficiently used. We also implemented new techniques which reduce the amount of work on the CPU side and the amount of communication between CPU and accelerators. We have measured the performance of N-body simulations on a system with an NVIDIA Volta GPGPU using FDPS and the achieved performance is around 27% of the theoretical peak limit. We have constructed a detailed performance model, and found that the current implementation can achieve good performance on systems with much smaller memory and communication bandwidth. Thus, our implementation will be applicable to future generations of accelerator system.

scholarly journals FQSqueezer: k-mer-based compression of sequencing data

2019 ◽  
Author(s):  
Sebastian Deorowicz
Keyword(s):  
Data Compression ◽  
State Of The Art ◽  
Genomic Data ◽  
General Purpose ◽  
Supplementary Information ◽  
Supplementary Data ◽  
Sequencing Data ◽  
Partial Matching ◽  
Supplementary Material ◽  
Better Than

AbstractMotivationThe amount of genomic data that needs to be stored is huge. Therefore it is not surprising that a lot of work has been done in the field of specialized data compression of FASTQ files. The existing algorithms are, however, still imperfect and the best tools produce quite large archives.ResultsWe present FQSqueezer, a novel compression algorithm for sequencing data able to process single- and paired-end reads of variable lengths. It is based on the ideas from the famous prediction by partial matching and dynamic Markov coder algorithms known from the general-purpose-compressors world. The compression ratios are often tens of percent better than offered by the state-of-the-art tools.Availability and Implementationhttps://github.com/refresh-bio/[email protected] informationSupplementary data are available at publisher’s Web site.

scholarly journals The Bio::Phylo libraries for phylogenetic data analysis, version 2.0

2017 ◽  
Author(s):  
Rutger A. Vos ◽  
Hannes Hettling
Keyword(s):  
Data Analysis ◽  
Open Source Software ◽  
Analysis Data ◽  
Data Exploration ◽  
Supplementary Information ◽  
Data Types ◽  
Phylogenetic Data ◽  
File Formats ◽  
Version 2.0 ◽  
Data Reading

AbstractMotivationPhylogenetic analysis is a broad and expanding field that requires versatile programming toolkits to manage the various data types, file formats, and needs for scalability, simulation, visualization, and data exploration.ResultsWe present version 2.0 of the Bio::Phylo libraries for phylogenetic data analysis. This new release represents a rewrite of the architecture, allowing for extensions that improve speed and persistence, as well as increased functionality in terms of analysis, data reading and writing, and visualization.AvailabilityThe package is released as open source software under the same terms as Perl itself and available from the comprehensive Perl archive network as well as directly from the source code [email protected] informationSupplementary data are available asdoi:10.5281/zenodo.1039210

scholarly journals hts-nim: scripting high-performance genomic analyses

2018 ◽  
Author(s):  
Brent S. Pedersen ◽  
Aaron R. Quinlan
Keyword(s):  
High Performance ◽  
Genomic Data ◽  
Supplementary Information ◽  
Supplementary Data ◽  
Scripting Languages ◽  
Link Type ◽  
Custom Software ◽  
Genomic Analyses ◽  
Biological Insight ◽  
Supplementary Material

AbstractMotivationExtracting biological insight from genomic data inevitably requires custom software. In many cases, this is accomplished with scripting languages, owing to their accessibility and brevity. Unfortunately, the ease of scripting languages typically comes at a substantial performance cost that is especially acute with the scale of modern genomics datasets.ResultsWe present hts-nim, a high-performance library written in the Nim programming language that provides a simple, scripting-like syntax without sacrificing performance.Availabilityhts-nim is available at https://github.com/brentp/hts-nim and the example tools are at https://github.com/brentp/hts-nim-tools both under the MIT [email protected] informationSupplementary data are available at Bioinformatics online.

Design Tradeoffs of High Performance DSPs for General-Purpose HPC

2014 ◽  
Vol 36 (4) ◽  
pp. 790-798
Author(s):  
Kai ZHANG ◽  
Shu-Ming CHEN ◽  
Yao-Hua WANG ◽  
Xi NING
Keyword(s):  
High Performance ◽  
General Purpose ◽  
Design Tradeoffs

scholarly journals DSPSR: Digital Signal Processing Software for Pulsar Astronomy

2011 ◽  
Vol 28 (1) ◽  
pp. 1-14 ◽  
Author(s):  
W. van Straten ◽  
M. Bailes
Keyword(s):  
Signal Processing ◽  
Digital Signal Processing ◽  
Graphics Processing Units ◽  
High Performance ◽  
Digital Signal ◽  
General Purpose ◽  
Design Decisions ◽  
Extensive Range ◽  
Processing Software ◽  
Graphics Processing

Abstractdspsr is a high-performance, open-source, object-oriented, digital signal processing software library and application suite for use in radio pulsar astronomy. Written primarily in C++, the library implements an extensive range of modular algorithms that can optionally exploit both multiple-core processors and general-purpose graphics processing units. After over a decade of research and development, dspsr is now stable and in widespread use in the community. This paper presents a detailed description of its functionality, justification of major design decisions, analysis of phase-coherent dispersion removal algorithms, and demonstration of performance on some contemporary microprocessor architectures.

scholarly journals BioCommons: a robust java library for RNA structural bioinformatics

2021 ◽  
Author(s):  
Tomasz Zok
Keyword(s):  
Source Code ◽  
Structural Bioinformatics ◽  
Supplementary Information ◽  
Supplementary Data ◽  
Bioinformatic Tools ◽  
Data Formats ◽  
Central Repository ◽  
Diverse Data ◽  
2D And 3D ◽  
Java Library

Abstract Motivation Biomolecular structures come in multiple representations and diverse data formats. Their incompatibility with the requirements of data analysis programs significantly hinders the analytics and the creation of new structure-oriented bioinformatic tools. Therefore, the need for robust libraries of data processing functions is still growing. Results BioCommons is an open-source, Java library for structural bioinformatics. It contains many functions working with the 2D and 3D structures of biomolecules, with a particular emphasis on RNA. Availability and implementation The library is available in Maven Central Repository and its source code is hosted on GitHub: https://github.com/tzok/BioCommons Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals GWASpro: a high-performance genome-wide association analysis server

2018 ◽  
Vol 35 (14) ◽  
pp. 2512-2514 ◽  
Author(s):  
Bongsong Kim ◽  
Xinbin Dai ◽  
Wenchao Zhang ◽  
Zhaohong Zhuang ◽  
Darlene L Sanchez ◽  
...  
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Linear Mixed Model ◽  
Association Studies ◽  
Learning Curves ◽  
Experimental Designs ◽  
Genome Wide Association ◽  
Supplementary Information ◽  
Genome Wide Association Studies ◽  
Genome Wide

Abstract Summary We present GWASpro, a high-performance web server for the analyses of large-scale genome-wide association studies (GWAS). GWASpro was developed to provide data analyses for large-scale molecular genetic data, coupled with complex replicated experimental designs such as found in plant science investigations and to overcome the steep learning curves of existing GWAS software tools. GWASpro supports building complex design matrices, by which complex experimental designs that may include replications, treatments, locations and times, can be accounted for in the linear mixed model. GWASpro is optimized to handle GWAS data that may consist of up to 10 million markers and 10 000 samples from replicable lines or hybrids. GWASpro provides an interface that significantly reduces the learning curve for new GWAS investigators. Availability and implementation GWASpro is freely available at https://bioinfo.noble.org/GWASPRO. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals TAIJI: approaching experimental replicates-level accuracy for drug synergy prediction

2018 ◽  
Vol 35 (13) ◽  
pp. 2338-2339 ◽  
Author(s):  
Hongyang Li ◽  
Shuai Hu ◽  
Nouri Neamati ◽  
Yuanfang Guan
Keyword(s):  
Drug Combination ◽  
High Performance ◽  
Synergistic Effects ◽  
Field Performance ◽  
Supplementary Information ◽  
Drug Synergism ◽  
High Throughput Drug Screening ◽  
Current State ◽  
Candidate Drug ◽  
High Prediction

Abstract Motivation Combination therapy is widely used in cancer treatment to overcome drug resistance. High-throughput drug screening is the standard approach to study the drug combination effects, yet it becomes impractical when the number of drugs under consideration is large. Therefore, accurate and fast computational tools for predicting drug synergistic effects are needed to guide experimental design for developing candidate drug pairs. Results Here, we present TAIJI, a high-performance software for fast and accurate prediction of drug synergism. It is based on the winning algorithm in the AstraZeneca-Sanger Drug Combination Prediction DREAM Challenge, which is a unique platform to unbiasedly evaluate the performance of current state-of-the-art methods, and includes 160 team-based submission methods. When tested across a broad spectrum of 85 different cancer cell lines and 1089 drug combinations, TAIJI achieved a high prediction correlation (0.53), approaching the accuracy level of experimental replicates (0.56). The runtime is at the scale of minutes to achieve this state-of-the-field performance. Availability and implementation TAIJI is freely available on GitHub (https://github.com/GuanLab/TAIJI). It is functional with built-in Perl and Python. Supplementary information Supplementary data are available at Bioinformatics online.

Sign in / Sign up

Export Citation Format

Share Document