FCompress: An Algorithm for FASTQ Sequence Data Compression

2019 ◽  
Vol 14 (2) ◽  
pp. 123-129
Author(s):  
Muhammad Sardaraz ◽  
Muhammad Tahir
Keyword(s):  
Data Compression ◽  
Data Storage ◽  
Sequence Data ◽  
Analysis Data ◽  
General Purpose ◽  
Huffman Coding ◽  
Biological Sequence ◽  
Sequencing Data ◽  
Transmission Cost ◽  
Capacity Data

Background: Biological sequence data have increased at a rapid rate due to the advancements in sequencing technologies and reduction in the cost of sequencing data. The huge increase in these data presents significant research challenges to researchers. In addition to meaningful analysis, data storage is also a challenge, an increase in data production is outpacing the storage capacity. Data compression is used to reduce the size of data and thus reduces storage requirements as well as transmission cost over the internet. Objective: This article presents a novel compression algorithm (FCompress) for Next Generation Sequencing (NGS) data in FASTQ format. Method: The proposed algorithm uses bits manipulation and dictionary-based compression for bases compression. Headers are compressed with reference-based compression, whereas quality scores are compressed with Huffman coding. Results: The proposed algorithm is validated with experimental results on real datasets. The results are compared with both general purpose and specialized compression programs. Conclusion: The proposed algorithm produces better compression ratio in a comparable time to other algorithms.

scholarly journals FQSqueezer: k-mer-based compression of sequencing data

2019 ◽  
Author(s):  
Sebastian Deorowicz
Keyword(s):  
Data Compression ◽  
State Of The Art ◽  
Genomic Data ◽  
General Purpose ◽  
Supplementary Information ◽  
Supplementary Data ◽  
Sequencing Data ◽  
Partial Matching ◽  
Supplementary Material ◽  
Better Than

AbstractMotivationThe amount of genomic data that needs to be stored is huge. Therefore it is not surprising that a lot of work has been done in the field of specialized data compression of FASTQ files. The existing algorithms are, however, still imperfect and the best tools produce quite large archives.ResultsWe present FQSqueezer, a novel compression algorithm for sequencing data able to process single- and paired-end reads of variable lengths. It is based on the ideas from the famous prediction by partial matching and dynamic Markov coder algorithms known from the general-purpose-compressors world. The compression ratios are often tens of percent better than offered by the state-of-the-art tools.Availability and Implementationhttps://github.com/refresh-bio/[email protected] informationSupplementary data are available at publisher’s Web site.

scholarly journals Streamlining Data-Intensive Biology With Workflow Systems

2020 ◽  
Author(s):  
Taylor Reiter ◽  
Phillip T. Brooks ◽  
Luiz Irber ◽  
Shannon E.K. Joslin ◽  
Charles M. Reid ◽  
...  
Keyword(s):  
Data Analysis ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
Sequence Data ◽  
Open Science ◽  
Biological Data ◽  
Data Generation ◽  
Biological Sequence ◽  
Sequencing Data ◽  
Workflow Systems

AbstractAs the scale of biological data generation has increased, the bottleneck of research has shifted from data generation to analysis. Researchers commonly need to build computational workflows that include multiple analytic tools and require incremental development as experimental insights demand tool and parameter modifications. These workflows can produce hundreds to thousands of intermediate files and results that must be integrated for biological insight. Data-centric workflow systems that internally manage computational resources, software, and conditional execution of analysis steps are reshaping the landscape of biological data analysis, and empowering researchers to conduct reproducible analyses at scale. Adoption of these tools can facilitate and expedite robust data analysis, but knowledge of these techniques is still lacking. Here, we provide a series of practices and strategies for leveraging workflow systems with structured project, data, and resource management to streamline large-scale biological analysis. We present these strategies in the context of high-throughput sequencing data analysis, but the principles are broadly applicable to biologists working beyond this field.Author SummaryWe present a guide for workflow-enabled biological sequence data analysis, developed through our own teaching, training and analysis projects. We recognize that this is based on our own use cases and experiences, but we hope that our guide will contribute to a larger discussion within the open source and open science communities and lead to more comprehensive resources. Our main goal is to accelerate the research of scientists conducting sequence analyses by introducing them to organized workflow practices that not only benefit their own research but also facilitate open and reproducible science.

scholarly journals SCA-NGS: Secure compression algorithm for next generation sequencing data using genetic operators and block sorting

2021 ◽  
Vol 104 (2) ◽  
pp. 003685042110232
Author(s):  
Muhammad Sardaraz ◽  
Muhammad Tahir
Keyword(s):  
Data Compression ◽  
State Of The Art ◽  
General Purpose ◽  
Next Generation Sequencing Data ◽  
Sequencing Data ◽  
Genetic Operators ◽  
Data Confidentiality ◽  
Research Challenges ◽  
Benchmark Datasets ◽  
Generation Sequencing

Recent advancements in sequencing methods have led to significant increase in sequencing data. Increase in sequencing data leads to research challenges such as storage, transfer, processing, etc. data compression techniques have been opted to cope with the storage of these data. There have been good achievements in compression ratio and execution time. This fast-paced advancement has raised major concerns about the security of data. Confidentiality, integrity, authenticity of data needs to be ensured. This paper presents a novel lossless reference-free algorithm that focuses on data compression along with encryption to achieve security in addition to other parameters. The proposed algorithm uses preprocessing of data before applying general-purpose compression library. Genetic algorithm is used to encrypt the data. The technique is validated with experimental results on benchmark datasets. Comparative analysis with state-of-the-art techniques is presented. The results show that the proposed method achieves better results in comparison to existing methods.

Advances in high throughput DNA sequence data compression

2016 ◽  
Vol 14 (03) ◽  
pp. 1630002 ◽  
Author(s):  
Muhammad Sardaraz ◽  
Muhammad Tahir ◽  
Ataul Aziz Ikram
Keyword(s):  
Data Compression ◽  
Dna Sequence ◽  
High Throughput ◽  
High Throughput Sequencing ◽  
Sequence Data ◽  
Sequencing Data ◽  
Research Directions ◽  
Compression Algorithms ◽  
Dna Sequence Data ◽  
Sequencing Technologies

Advances in high throughput sequencing technologies and reduction in cost of sequencing have led to exponential growth in high throughput DNA sequence data. This growth has posed challenges such as storage, retrieval, and transmission of sequencing data. Data compression is used to cope with these challenges. Various methods have been developed to compress genomic and sequencing data. In this article, we present a comprehensive review of compression methods for genome and reads compression. Algorithms are categorized as referential or reference free. Experimental results and comparative analysis of various methods for data compression are presented. Finally, key challenges and research directions in DNA sequence data compression are highlighted.

scholarly journals Design and development of learning model for compression and processing of deoxyribonucleic acid genome sequence

2022 ◽  
Vol 12 (2) ◽  
pp. 1786
Author(s):  
Raveendra Gudodagi ◽  
Rayapur Venkata Siva Reddy ◽  
Mohammed Riyaz Ahmed
Keyword(s):  
Deep Learning ◽  
Data Compression ◽  
Data Storage ◽  
Genome Sequence ◽  
Sequence Data ◽  
Genomic Data ◽  
Modern Technology ◽  
Optimization Techniques ◽  
Learning System ◽  
Systematic Analysis

Owing to the substantial volume of human genome sequence data files (from 30-200 GB exposed) Genomic data compression has received considerable traction and storage costs are one of the major problems faced by genomics laboratories. This involves a modern technology of data compression that reduces not only the storage but also the reliability of the operation. There were few attempts to solve this problem independently of both hardware and software. A systematic analysis of associations between genes provides techniques for the recognition of operative connections among genes and their respective yields, as well as understandings into essential biological events that are most important for knowing health and disease phenotypes. This research proposes a reliable and efficient deep learning system for learning embedded projections to combine gene interactions and gene expression in prediction comparison of deep embeddings to strong baselines. In this paper we preform data processing operations and predict gene function, along with gene ontology reconstruction and predict the gene interaction. The three major steps of genomic data compression are extraction of data, storage of data, and retrieval of the data. Hence, we propose a deep learning based on computational optimization techniques which will be efficient in all the three stages of data compression.

scholarly journals A Lossless Compression Method for Chat Messages Based on Huffman Coding and Dynamic Programming

Computers ◽  
2021 ◽  
Vol 10 (3) ◽  
pp. 28
Author(s):  
Julián Moreno Cadavid ◽  
Hernán Darío Vanegas Madrigal
Keyword(s):  
Dynamic Programming ◽  
Data Compression ◽  
Data Storage ◽  
Greedy Algorithms ◽  
Short Length ◽  
Huffman Coding ◽  
Compression Method ◽  
Average Value ◽  
Increasing Demand ◽  
Algorithmic Approaches

There is always an increasing demand for data storage and transfer; therefore, data compression will always be a fundamental need. In this article, we propose a lossless data compression method focused on a particular kind of data, namely, chat messages, which are typically non-formal, short-length strings. This method can be considered a hybrid because it combines two different algorithmic approaches: greedy algorithms, specifically Huffman coding, on the one hand and dynamic programming on the other (HCDP = Huffman Coding + Dynamic Programming). The experimental results demonstrated that our method provided lower compression ratios when compared with six reference algorithms, with reductions between 23.7% and 39.7%, whilst the average remained below the average value reported in several related works found in the literature. Such performance carries a sacrifice in speed, however, which does not presume major practical implications in the context of short-length strings.

Application of an image management system in microscopy

1992 ◽  
Vol 50 (2) ◽  
pp. 1052-1053
Author(s):  
Richard S. Chemock
Keyword(s):  
Data Storage ◽  
Cost Savings ◽  
Image Data ◽  
Analytical Techniques ◽  
Operational Mode ◽  
Fine Grained ◽  
Image Recording ◽  
Primary Output ◽  
Capacity Data ◽  
Image Management System

One of the most common tasks in a typical analysis lab is the recording of images. Many analytical techniques (TEM, SEM, and metallography for example) produce images as their primary output. Until recently, the most common method of recording images was by using film. Current PS/2R systems offer very large capacity data storage devices and high resolution displays, making it practical to work with analytical images on PS/2s, thereby sidestepping the traditional film and darkroom steps. This change in operational mode offers many benefits: cost savings, throughput, archiving and searching capabilities as well as direct incorporation of the image data into reports.The conventional way to record images involves film, either sheet film (with its associated wet chemistry) for TEM or PolaroidR film for SEM and light microscopy. Although film is inconvenient, it does have the highest quality of all available image recording techniques. The fine grained film used for TEM has a resolution that would exceed a 4096x4096x16 bit digital image.

Design and Implementation of Low Energy Wireless Network Nodes based on Hardware Compression Acceleration

2019 ◽  
Vol 12 ◽  
Author(s):  
Hui Yang ◽  
Anand Nayyar
Keyword(s):  
Energy Consumption ◽  
Data Compression ◽  
Energy Saving ◽  
Optimization Design ◽  
Hardware Acceleration ◽  
Transmission Efficiency ◽  
General Purpose ◽  
Storage Space ◽  
General Purpose Processor ◽  
Compression Time

: In the fast development of information, the information data is increasing in geometric multiples, and the speed of information transmission and storage space are required to be higher. In order to reduce the use of storage space and further improve the transmission efficiency of data, data need to be compressed. processing. In the process of data compression, it is very important to ensure the lossless nature of data, and lossless data compression algorithms appear. The gradual optimization design of the algorithm can often achieve the energy-saving optimization of data compression. Similarly, The effect of energy saving can also be obtained by improving the hardware structure of node. In this paper, a new structure is designed for sensor node, which adopts hardware acceleration, and the data compression module is separated from the node microprocessor.On the basis of the ASIC design of the algorithm, by introducing hardware acceleration, the energy consumption of the compressed data was successfully reduced, and the proportion of energy consumption and compression time saved by the general-purpose processor was as high as 98.4 % and 95.8 %, respectively. It greatly reduces the compression time and energy consumption.

scholarly journals AC2: An Efficient Protein Sequence Compression Tool Using Artificial Neural Networks and Cache-Hash Models

Entropy ◽  
2021 ◽  
Vol 23 (5) ◽  
pp. 530
Author(s):  
Milton Silva ◽  
Diogo Pratas ◽  
Armando J. Pinho
Keyword(s):  
Protein Sequence ◽  
Sequence Data ◽  
Specific Protein ◽  
General Purpose ◽  
Amino Acid Sequences ◽  
Input Size ◽  
Protein Sequence Data ◽  
Analysis Application ◽  
Straightforward Solution ◽  
Human Coronaviruses

Recently, the scientific community has witnessed a substantial increase in the generation of protein sequence data, triggering emergent challenges of increasing importance, namely efficient storage and improved data analysis. For both applications, data compression is a straightforward solution. However, in the literature, the number of specific protein sequence compressors is relatively low. Moreover, these specialized compressors marginally improve the compression ratio over the best general-purpose compressors. In this paper, we present AC2, a new lossless data compressor for protein (or amino acid) sequences. AC2 uses a neural network to mix experts with a stacked generalization approach and individual cache-hash memory models to the highest-context orders. Compared to the previous compressor (AC), we show gains of 2–9% and 6–7% in reference-free and reference-based modes, respectively. These gains come at the cost of three times slower computations. AC2 also improves memory usage against AC, with requirements about seven times lower, without being affected by the sequences’ input size. As an analysis application, we use AC2 to measure the similarity between each SARS-CoV-2 protein sequence with each viral protein sequence from the whole UniProt database. The results consistently show higher similarity to the pangolin coronavirus, followed by the bat and human coronaviruses, contributing with critical results to a current controversial subject. AC2 is available for free download under GPLv3 license.

scholarly journals REscan: inferring repeat expansions and structural variation in paired-end short read sequencing data

2020 ◽  
Author(s):  
Russell Lewis McLaughlin
Keyword(s):  
Structural Variation ◽  
Sequence Data ◽  
Neurological Diseases ◽  
Repeat Expansion ◽  
Sequencing Data ◽  
Short Read ◽  
Short Read Sequencing ◽  
Repeat Expansions ◽  
Paired End Sequencing

Abstract Motivation Repeat expansions are an important class of genetic variation in neurological diseases. However, the identification of novel repeat expansions using conventional sequencing methods is a challenge due to their typical lengths relative to short sequence reads and difficulty in producing accurate and unique alignments for repetitive sequence. However, this latter property can be harnessed in paired-end sequencing data to infer the possible locations of repeat expansions and other structural variation. Results This article presents REscan, a command-line utility that infers repeat expansion loci from paired-end short read sequencing data by reporting the proportion of reads orientated towards a locus that do not have an adequately mapped mate. A high REscan statistic relative to a population of data suggests a repeat expansion locus for experimental follow-up. This approach is validated using genome sequence data for 259 cases of amyotrophic lateral sclerosis, of which 24 are positive for a large repeat expansion in C9orf72, showing that REscan statistics readily discriminate repeat expansion carriers from non-carriers. Availabilityand implementation C source code at https://github.com/rlmcl/rescan (GNU General Public Licence v3).

Sign in / Sign up

Export Citation Format

Share Document