scholarly journals Streamlining Data-Intensive Biology With Workflow Systems

2020 ◽  
Author(s):  
Taylor Reiter ◽  
Phillip T. Brooks ◽  
Luiz Irber ◽  
Shannon E.K. Joslin ◽  
Charles M. Reid ◽  
...  
Keyword(s):  
Data Analysis ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
Sequence Data ◽  
Open Science ◽  
Biological Data ◽  
Data Generation ◽  
Biological Sequence ◽  
Sequencing Data ◽  
Workflow Systems

AbstractAs the scale of biological data generation has increased, the bottleneck of research has shifted from data generation to analysis. Researchers commonly need to build computational workflows that include multiple analytic tools and require incremental development as experimental insights demand tool and parameter modifications. These workflows can produce hundreds to thousands of intermediate files and results that must be integrated for biological insight. Data-centric workflow systems that internally manage computational resources, software, and conditional execution of analysis steps are reshaping the landscape of biological data analysis, and empowering researchers to conduct reproducible analyses at scale. Adoption of these tools can facilitate and expedite robust data analysis, but knowledge of these techniques is still lacking. Here, we provide a series of practices and strategies for leveraging workflow systems with structured project, data, and resource management to streamline large-scale biological analysis. We present these strategies in the context of high-throughput sequencing data analysis, but the principles are broadly applicable to biologists working beyond this field.Author SummaryWe present a guide for workflow-enabled biological sequence data analysis, developed through our own teaching, training and analysis projects. We recognize that this is based on our own use cases and experiences, but we hope that our guide will contribute to a larger discussion within the open source and open science communities and lead to more comprehensive resources. Our main goal is to accelerate the research of scientists conducting sequence analyses by introducing them to organized workflow practices that not only benefit their own research but also facilitate open and reproducible science.

scholarly journals Streamlining data-intensive biology with workflow systems

GigaScience ◽  
2021 ◽  
Vol 10 (1) ◽  
Author(s):  
Taylor Reiter ◽  
Phillip T Brooks† ◽  
Luiz Irber† ◽  
Shannon E K Joslin† ◽  
Charles M Reid† ◽  
...  
Keyword(s):  
Data Analysis ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
Biological Data ◽  
Data Generation ◽  
Sequencing Data ◽  
Workflow Systems ◽  
Data Intensive ◽  
High Throughput Sequencing Data ◽  
Project Data

Abstract As the scale of biological data generation has increased, the bottleneck of research has shifted from data generation to analysis. Researchers commonly need to build computational workflows that include multiple analytic tools and require incremental development as experimental insights demand tool and parameter modifications. These workflows can produce hundreds to thousands of intermediate files and results that must be integrated for biological insight. Data-centric workflow systems that internally manage computational resources, software, and conditional execution of analysis steps are reshaping the landscape of biological data analysis and empowering researchers to conduct reproducible analyses at scale. Adoption of these tools can facilitate and expedite robust data analysis, but knowledge of these techniques is still lacking. Here, we provide a series of strategies for leveraging workflow systems with structured project, data, and resource management to streamline large-scale biological analysis. We present these practices in the context of high-throughput sequencing data analysis, but the principles are broadly applicable to biologists working beyond this field.

scholarly journals CRAFT: Compact genome Representation toward large-scale Alignment-Free daTabase

2020 ◽  
Author(s):  
Yang Young Lu ◽  
Jiaxing Bai ◽  
Yiwen Wang ◽  
Ying Wang ◽  
Fengzhu Sun
Keyword(s):  
Dna Sequences ◽  
Sequence Comparison ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
Sequence Data ◽  
Practical Interest ◽  
Supplementary Information ◽  
Sequencing Data ◽  
Computationally Efficient ◽  
Alignment Free

Abstract Motivation Rapid developments in sequencing technologies have boosted generating high volumes of sequence data. To archive and analyze those data, one primary step is sequence comparison. Alignment-free sequence comparison based on k-mer frequencies offers a computationally efficient solution, yet in practice, the k-mer frequency vectors for large k of practical interest lead to excessive memory and storage consumption. Results We report CRAFT, a general genomic/metagenomic search engine to learn compact representations of sequences and perform fast comparison between DNA sequences. Specifically, given genome or high throughput sequencing data as input, CRAFT maps the data into a much smaller embedding space and locates the best matching genome in the archived massive sequence repositories. With 102−104-fold reduction of storage space, CRAFT performs fast query for gigabytes of data within seconds or minutes, achieving comparable performance as six state-of-the-art alignment-free measures. Availability and implementation CRAFT offers a user-friendly graphical user interface with one-click installation on Windows and Linux operating systems, freely available at https://github.com/jiaxingbai/CRAFT. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals An Open-Source Toolkit To Expand Bioinformatics Training in Infectious Diseases

mBio ◽  
2021 ◽  
Author(s):  
Alexander S. F. Berry ◽  
Camila Farias Amorim ◽  
Corbett L. Berry ◽  
Camille M. Syrett ◽  
Elise D. English ◽  
...  
Keyword(s):  
Infectious Disease ◽  
Data Analysis ◽  
High Throughput Sequencing ◽  
Data Sets ◽  
Data Generation ◽  
Sequencing Data ◽  
Sequencing Technology ◽  
Didactic Instruction ◽  
Host Parasite Interactions ◽  
Host Parasite

As access to high-throughput sequencing technology has increased, the bottleneck in biomedical research has shifted from data generation to data analysis. Here, we describe a modular and extensible framework for didactic instruction in bioinformatics using publicly available RNA sequencing data sets from infectious disease studies, with a focus on host-parasite interactions.

scholarly journals DDBJ update: streamlining submission and access of human data

2020 ◽  
Vol 49 (D1) ◽  
pp. D71-D75
Author(s):  
Asami Fukuda ◽  
Yuichi Kodama ◽  
Jun Mashima ◽  
Takatomo Fujisawa ◽  
Osamu Ogasawara
Keyword(s):  
Large Scale ◽  
High Throughput Sequencing ◽  
Group Structure ◽  
Biological Data ◽  
Sequencing Data ◽  
The Public ◽  
Human Data ◽  
Phenotype Data ◽  
Sequencing Platforms ◽  
Authorized Access

Abstract The Bioinformation and DDBJ Center (DDBJ Center, https://www.ddbj.nig.ac.jp) provides databases that capture, preserve and disseminate diverse biological data to support research in the life sciences. This center collects nucleotide sequences with annotations, raw sequencing data, and alignment information from high-throughput sequencing platforms, and study and sample information, in collaboration with the National Center for Biotechnology Information (NCBI) and the European Bioinformatics Institute (EBI). This collaborative framework is known as the International Nucleotide Sequence Database Collaboration (INSDC). In collaboration with the National Bioscience Database Center (NBDC), the DDBJ Center also provides a controlled-access database, the Japanese Genotype–phenotype Archive (JGA), which archives and distributes human genotype and phenotype data, requiring authorized access. The NBDC formulates guidelines and policies for sharing human data and reviews data submission and use applications. To streamline all of the processes at NBDC and JGA, we have integrated the two systems by introducing a unified login platform with a group structure in September 2020. In addition to the public databases, the DDBJ Center provides a computer resource, the NIG supercomputer, for domestic researchers to analyze large-scale genomic data. This report describes updates to the services of the DDBJ Center, focusing on the NBDC and JGA system enhancements.

scholarly journals Improvements and impacts of GRCh38 human reference on high throughput sequencing data analysis

Genomics ◽  
2017 ◽  
Vol 109 (2) ◽  
pp. 83-90 ◽  
Author(s):  
Yan Guo ◽  
Yulin Dai ◽  
Hui Yu ◽  
Shilin Zhao ◽  
David C. Samuels ◽  
...  
Keyword(s):  
Data Analysis ◽  
High Throughput ◽  
High Throughput Sequencing ◽  
Sequencing Data ◽  
High Throughput Sequencing Data ◽  
Sequencing Data Analysis

scholarly journals Utilizing the VirIdAl Pipeline to Search for Viruses in the Metagenomic Data of Bat Samples

Viruses ◽  
2021 ◽  
Vol 13 (10) ◽  
pp. 2006
Author(s):  
Anna Y Budkina ◽  
Elena V Korneenko ◽  
Ivan A Kotov ◽  
Daniil A Kiselev ◽  
Ilya V Artyushin ◽  
...  
Keyword(s):  
Large Scale ◽  
High Throughput Sequencing ◽  
Metagenomic Data ◽  
Sequencing Data ◽  
Viral Pathogens ◽  
Genomic Databases ◽  
Bioinformatic Pipeline ◽  
Viral Genomes ◽  
Sequencing Technologies ◽  
Viral Screening

According to various estimates, only a small percentage of existing viruses have been discovered, naturally much less being represented in the genomic databases. High-throughput sequencing technologies develop rapidly, empowering large-scale screening of various biological samples for the presence of pathogen-associated nucleotide sequences, but many organisms are yet to be attributed specific loci for identification. This problem particularly impedes viral screening, due to vast heterogeneity in viral genomes. In this paper, we present a new bioinformatic pipeline, VirIdAl, for detecting and identifying viral pathogens in sequencing data. We also demonstrate the utility of the new software by applying it to viral screening of the feces of bats collected in the Moscow region, which revealed a significant variety of viruses associated with bats, insects, plants, and protozoa. The presence of alpha and beta coronavirus reads, including the MERS-like bat virus, deserves a special mention, as it once again indicates that bats are indeed reservoirs for many viral pathogens. In addition, it was shown that alignment-based methods were unable to identify the taxon for a large proportion of reads, and we additionally applied other approaches, showing that they can further reveal the presence of viral agents in sequencing data. However, the incompleteness of viral databases remains a significant problem in the studies of viral diversity, and therefore necessitates the use of combined approaches, including those based on machine learning methods.

scholarly journals Tree-Aggregated Predictive Modeling of Microbiome Data

2020 ◽  
Author(s):  
Jacob Bien ◽  
Xiaohan Yan ◽  
Léo Simpson ◽  
Christian L. Müller
Keyword(s):  
Data Analysis ◽  
Predictive Modeling ◽  
Large Scale ◽  
High Throughput Sequencing ◽  
Compositional Data ◽  
Low Cost ◽  
Primary Data ◽  
Compositional Data Analysis ◽  
Taxonomic Rank ◽  
Microbiome Data

AbstractModern high-throughput sequencing technologies provide low-cost microbiome survey data across all habitats of life at unprecedented scale. At the most granular level, the primary data consist of sparse counts of amplicon sequence variants or operational taxonomic units that are associated with taxonomic and phylogenetic group information. In this contribution, we leverage the hierarchical structure of amplicon data and propose a data-driven, parameter-free, and scalable tree-guided aggregation framework to associate microbial subcompositions with response variables of interest. The excess number of zero or low count measurements at the read level forces traditional microbiome data analysis workflows to remove rare sequencing variants or group them by a fixed taxonomic rank, such as genus or phylum, or by phylogenetic similarity. By contrast, our framework, which we call trac (tree-aggregation of compositional data), learns data-adaptive taxon aggregation levels for predictive modeling making user-defined aggregation obsolete while simultaneously integrating seamlessly into the compositional data analysis framework. We illustrate the versatility of our framework in the context of large-scale regression problems in human-gut, soil, and marine microbial ecosystems. We posit that the inferred aggregation levels provide highly interpretable taxon groupings that can help microbial ecologists gain insights into the structure and functioning of the underlying ecosystem of interest.

scholarly journals Inferring species compositions of complex fungal communities from long- and short-read sequence data

2021 ◽  
Author(s):  
Yiheng Hu ◽  
Laszlo Irinyi ◽  
Minh Thuy Vi Hoang ◽  
Tavish Eenjes ◽  
Abigail Graetz ◽  
...  
Keyword(s):  
Community Composition ◽  
Pathogen Detection ◽  
High Throughput Sequencing ◽  
Sequence Data ◽  
Whole Genome Sequence ◽  
Composition Analysis ◽  
Sequencing Data ◽  
Species Classification ◽  
Shotgun Metagenomics ◽  
Query Coverage

Background: The kingdom fungi is crucial for life on earth and is highly diverse. Yet fungi are challenging to characterize. They can be difficult to culture and may be morphologically indistinct in culture. They can have complex genomes of over 1 Gb in size and are still underrepresented in whole genome sequence databases. Overall their description and analysis lags far behind other microbes such as bacteria. At the same time, classification of species via high throughput sequencing without prior purification is increasingly becoming the norm for pathogen detection, microbiome studies, and environmental monitoring. However, standardized procedures for characterizing unknown fungi from complex sequencing data have not yet been established. Results: We compared different metagenomics sequencing and analysis strategies for the identification of fungal species. Using two fungal mock communities of 44 phylogenetically diverse species, we compared species classification and community composition analysis pipelines using shotgun metagenomics and amplicon sequencing data generated from both short and long read sequencing technologies. We show that regardless of the sequencing methodology used, the highest accuracy of species identification was achieved by sequence alignment against a fungi-specific database. During the assessment of classification algorithms, we found that applying cut-offs to the query coverage of each read or contig significantly improved the classification accuracy and community composition analysis without significant data loss. Conclusion: Overall, our study expands the toolkit for identifying fungi by improving sequence-based fungal classification, and provides a practical guide for the design of metagenomics analyses.

EdClust: A heuristic sequence clustering method with higher sensitivity

2021 ◽  
Author(s):  
Ming Cao ◽  
Qinke Peng ◽  
Ze-Gang Wei ◽  
Fei Liu ◽  
Yi-Fan Hou
Keyword(s):  
Large Scale ◽  
Sequence Data ◽  
Clustering Algorithms ◽  
Clustering Methods ◽  
Sequencing Data ◽  
Clustering Method ◽  
Cluster Number ◽  
Sequence Clustering ◽  
Downstream Analysis ◽  
Heuristic Clustering

The development of high-throughput technologies has produced increasing amounts of sequence data and an increasing need for efficient clustering algorithms that can process massive volumes of sequencing data for downstream analysis. Heuristic clustering methods are widely applied for sequence clustering because of their low computational complexity. Although numerous heuristic clustering methods have been developed, they suffer from two limitations: overestimation of inferred clusters and low clustering sensitivity. To address these issues, we present a new sequence clustering method (edClust) based on Edlib, a C/C[Formula: see text] library for fast, exact semi-global sequence alignment to group similar sequences. The new method edClust was tested on three large-scale sequence databases, and we compared edClust to several classic heuristic clustering methods, such as UCLUST, CD-HIT, and VSEARCH. Evaluations based on the metrics of cluster number and seed sensitivity (SS) demonstrate that edClust can produce fewer clusters than other methods and that its SS is higher than that of other methods. The source codes of edClust are available from https://github.com/zhang134/EdClust.git under the GNU GPL license.

Phylogenomics of orchids and their mycorrhizal fungi : trees, diversity, and the pursuit of symbiosis

2019 ◽  
Author(s):  
◽  
Sarah Unruh
Keyword(s):  
Mycorrhizal Fungi ◽  
Phylogenetic Trees ◽  
High Throughput Sequencing ◽  
Genomic Sequence ◽  
Sequence Data ◽  
Mycorrhizal Symbiosis ◽  
Sequencing Data ◽  
Phylogenetic Structure ◽  
University Of Missouri ◽  
Fungal Symbiosis

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT REQUEST OF AUTHOR.] Phylogenetic trees show us how organisms are related and provide frameworks for studying and testing evolutionary hypotheses. To better understand the evolution of orchids and their mycorrhizal fungi, I used high-throughput sequencing data and bioinformatic analyses, to build phylogenetic hypotheses. In Chapter 2, I used transcriptome sequences to both build a phylogeny of the slipper orchid genera and to confirm the placement of a polyploidy event at the base of the orchid family. Polyploidy is hypothesized to be a strong driver of evolution and a source of unique traits so confirming this event leads us closer to explaining extant orchid diversity. The list of orthologous genes generated from this study will provide a less expensive and more powerful method for researchers examining the evolutionary relationships in Orchidaceae. In Chapter 3, I generated genomic sequence data for 32 fungal isolates that were collected from orchids across North America. I inferred the first multi-locus nuclear phylogenetic tree for these fungal clades. The phylogenetic structure of these fungi will improve the taxonomy of these clades by providing evidence for new species and for revising problematic species designations. A robust taxonomy is necessary for studying the role of fungi in the orchid mycorrhizal symbiosis. In chapter 4 I summarize my work and outline the future directions of my lab at Illinois College including addressing the remaining aims of my Community Sequencing Proposal with the Joint Genome Institute by analyzing the 15 fungal reference genomes I generated during my PhD. Together these chapters are the start of a life-long research project into the evolution and function of the orchid/fungal symbiosis.

Sign in / Sign up

Export Citation Format

Share Document