scholarly journals DOE JGI Metagenome Workflow

mSystems ◽  
2021 ◽  
Vol 6 (3) ◽  
Author(s):  
Alicia Clum ◽  
Marcel Huntemann ◽  
Brian Bushnell ◽  
Brian Foster ◽  
Bryce Foster ◽  
...  
Keyword(s):  
Nucleic Acids ◽  
Data Processing ◽  
Microbial Communities ◽  
Taxonomic Composition ◽  
Metagenomic Data ◽  
Data Sets ◽  
Microbial Genomes ◽  
Link Type ◽  
Analysis System ◽  
Rich Data

ABSTRACT The DOE Joint Genome Institute (JGI) Metagenome Workflow performs metagenome data processing, including assembly; structural, functional, and taxonomic annotation; and binning of metagenomic data sets that are subsequently included into the Integrated Microbial Genomes and Microbiomes (IMG/M) (I.-M. A. Chen, K. Chu, K. Palaniappan, A. Ratner, et al., Nucleic Acids Res, 49:D751–D763, 2021, https://doi.org/10.1093/nar/gkaa939) comparative analysis system and provided for download via the JGI data portal (https://genome.jgi.doe.gov/portal/). This workflow scales to run on thousands of metagenome samples per year, which can vary by the complexity of microbial communities and sequencing depth. Here, we describe the different tools, databases, and parameters used at different steps of the workflow to help with the interpretation of metagenome data available in IMG and to enable researchers to apply this workflow to their own data. We use 20 publicly available sediment metagenomes to illustrate the computing requirements for the different steps and highlight the typical results of data processing. The workflow modules for read filtering and metagenome assembly are available as a workflow description language (WDL) file (https://code.jgi.doe.gov/BFoster/jgi_meta_wdl). The workflow modules for annotation and binning are provided as a service to the user community at https://img.jgi.doe.gov/submit and require filling out the project and associated metadata descriptions in the Genomes OnLine Database (GOLD) (S. Mukherjee, D. Stamatis, J. Bertsch, G. Ovchinnikova, et al., Nucleic Acids Res, 49:D723–D733, 2021, https://doi.org/10.1093/nar/gkaa983). IMPORTANCE The DOE JGI Metagenome Workflow is designed for processing metagenomic data sets starting from Illumina fastq files. It performs data preprocessing, error correction, assembly, structural and functional annotation, and binning. The results of processing are provided in several standard formats, such as fasta and gff, and can be used for subsequent integration into the Integrated Microbial Genomes and Microbiomes (IMG/M) system where they can be compared to a comprehensive set of publicly available metagenomes. As of 30 July 2020, 7,155 JGI metagenomes have been processed by the DOE JGI Metagenome Workflow. Here, we present a metagenome workflow developed at the JGI that generates rich data in standard formats and has been optimized for downstream analyses ranging from assessment of the functional and taxonomic composition of microbial communities to genome-resolved metagenomics and the identification and characterization of novel taxa. This workflow is currently being used to analyze thousands of metagenomic data sets in a consistent and standardized manner.

scholarly journals The DOE JGI Metagenome Workflow

2020 ◽  
Author(s):  
Alicia Clum ◽  
Marcel Huntemann ◽  
Brian Bushnell ◽  
Brian Foster ◽  
Bryce Foster ◽  
...  
Keyword(s):  
Nucleic Acids ◽  
Data Processing ◽  
Description Language ◽  
Microbial Genomes ◽  
Processing Error ◽  
Link Type ◽  
Metagenome Assembly ◽  
Data Portal ◽  
Analysis System ◽  
Taxonomic Annotation

ABSTRACTThe DOE JGI Metagenome Workflow performs metagenome data processing, including assembly, structural, functional, and taxonomic annotation, and binning of metagenomic datasets that are subsequently included into the Integrated Microbial Genomes and Microbiomes (IMG/M) comparative analysis system (I. Chen, K. Chu, K. Palaniappan, M. Pillay, A. Ratner, J. Huang, M. Huntemann, N. Varghese, J. White, R. Seshadri, et al, Nucleic Acids Rsearch, 2019) and provided for download via the Joint Genome Institute (JGI) Data Portal (https://genome.jgi.doe.gov/portal/). This workflow scales to run on thousands of metagenome samples per year, which can vary by the complexity of microbial communities and sequencing depth. Here we describe the different tools, databases, and parameters used at different steps of the workflow, to help with interpretation of metagenome data available in IMG and to enable researchers to apply this workflow to their own data. We use 20 publicly available sediment metagenomes to illustrate the computing requirements for the different steps and highlight the typical results of data processing. The workflow modules for read filtering and metagenome assembly are available as a Workflow Description Language (WDL) file (https://code.jgi.doe.gov/BFoster/jgi_meta_wdl.git). The workflow modules for annotation and binning are provided as a service to the user community at https://img.jgi.doe.gov/submit and require filling out the project and associated metadata descriptions in Genomes OnLine Database (GOLD) (S. Mukherjee, D. Stamatis, J. Bertsch, G. Ovchinnikova, H. Katta, A. Mojica, I Chen, and N. Kyrpides, and T. Reddy, Nucleic Acids Research, 2018).IMPORTANCEThe DOE JGI Metagenome Workflow is designed for processing metagenomic datasets starting from Illumina fastq files. It performs data pre-processing, error correction, assembly, structural and functional annotation, and binning. The results of processing are provided in several standard formats, such as fasta and gff and can be used for subsequent integration into the Integrated Microbial Genome (IMG) system where they can be compared to a comprehensive set of publicly available metagenomes. As of 7/30/2020 7,155 JGI metagenomes have been processed by the JGI Metagenome Workflow.

scholarly journals The impact of sequencing depth on the inferred taxonomic composition and AMR gene content of metagenomic samples

2019 ◽  
Author(s):  
H. Soon Gweon ◽  
Liam P. Shaw ◽  
Jeremy Swann ◽  
Nicola De Maio ◽  
Manal AbuOun ◽  
...  
Keyword(s):  
Data Processing ◽  
Open Source ◽  
Open Source Software ◽  
River Sediment ◽  
Taxonomic Composition ◽  
Sequencing Depth ◽  
Gene Content ◽  
Metagenomic Data ◽  
Software Pipeline ◽  
Shotgun Metagenomics

ABSTRACTBackgroundShotgun metagenomics is increasingly used to characterise microbial communities, particularly for the investigation of antimicrobial resistance (AMR) in different animal and environmental contexts. There are many different approaches for inferring the taxonomic composition and AMR gene content of complex community samples from shotgun metagenomic data, but there has been little work establishing the optimum sequencing depth, data processing and analysis methods for these samples. In this study we used shotgun metagenomics and sequencing of cultured isolates from the same samples to address these issues. We sampled three potential environmental AMR gene reservoirs (pig caeca, river sediment, effluent) and sequenced samples with shotgun metagenomics at high depth (∼200 million reads per sample). Alongside this, we cultured single-colony isolates ofEnterobacteriaceaefrom the same samples and used hybrid sequencing (short- and long-reads) to create high-quality assemblies for comparison to the metagenomic data. To automate data processing, we developed an open-source software pipeline, ‘ResPipe’.ResultsTaxonomic profiling was much more stable to sequencing depth than AMR gene content. 1 million reads per sample was sufficient to achieve <1% dissimilarity to the full taxonomic composition. However, at least 80 million reads per sample were required to recover the full richness of different AMR gene families present in the sample, and additional allelic diversity of AMR genes was still being discovered in effluent at 200 million reads per sample. Normalising the number of reads mapping to AMR genes using gene length and an exogenous spike ofThermus thermophilusDNA substantially changed the estimated gene abundance distributions. While the majority of genomic content from cultured isolates from effluent was recoverable using shotgun metagenomics, this was not the case for pig caeca or river sediment.ConclusionsSequencing depth and profiling method can critically affect the profiling of polymicrobial animal and environmental samples with shotgun metagenomics. Both sequencing of cultured isolates and shotgun metagenomics can recover substantial diversity that is not identified using the other methods. Particular consideration is required when inferring AMR gene content or presence by mapping metagenomic reads to a database. ResPipe, the open-source software pipeline we have developed, is freely available (https://gitlab.com/hsgweon/ResPipe).

scholarly journals Microbiome Search Engine 2: a Platform for Taxonomic and Functional Search of Global Microbiomes on the Whole-Microbiome Level

mSystems ◽  
2021 ◽  
Vol 6 (1) ◽  
Author(s):  
Gongchao Jing ◽  
Lu Liu ◽  
Zengbin Wang ◽  
Yufeng Zhang ◽  
Li Qian ◽  
...  
Keyword(s):  
Big Data ◽  
User Interface ◽  
Search Engine ◽  
Functional Similarity ◽  
Metagenomic Data ◽  
Data Sets ◽  
Data Space ◽  
Link Type ◽  
Database Platform ◽  
Microbiome Data

ABSTRACT Metagenomic data sets from diverse environments have been growing rapidly. To ensure accessibility and reusability, tools that quickly and informatively correlate new microbiomes with existing ones are in demand. Here, we introduce Microbiome Search Engine 2 (MSE 2), a microbiome database platform for searching query microbiomes in the global metagenome data space based on the taxonomic or functional similarity of a whole microbiome to those in the database. MSE 2 consists of (i) a well-organized and regularly updated microbiome database that currently contains over 250,000 metagenomic shotgun and 16S rRNA gene amplicon samples associated with unified metadata collected from 798 studies, (ii) an enhanced search engine that enables real-time and fast (<0.5 s per query) searches against the entire database for best-matched microbiomes using overall taxonomic or functional profiles, and (iii) a Web-based graphical user interface for user-friendly searching, data browsing, and tutoring. MSE 2 is freely accessible via http://mse.ac.cn. For standalone searches of customized microbiome databases, the kernel of the MSE 2 search engine is provided at GitHub (https://github.com/qibebt-bioinfo/meta-storms). IMPORTANCE A search-based strategy is useful for large-scale mining of microbiome data sets, such as a bird’s-eye view of the microbiome data space and disease diagnosis via microbiome big data. Here, we introduce Microbiome Search Engine 2 (MSE 2), a microbiome database platform for searching query microbiomes against the existing microbiome data sets on the basis of their similarity in taxonomic structure or functional profile. Key improvements include database extension, data compatibility, a search engine kernel, and a user interface. The new ability to search the microbiome space via functional similarity greatly expands the scope of search-based mining of the microbiome big data.

scholarly journals Bayesian Classification of Microbial Communities Based on 16S rRNA Metagenomic Data

2018 ◽  
Author(s):  
Arghavan Bahadorinejad ◽  
Ivan Ivanov ◽  
Johanna W Lampe ◽  
Meredith AJ Hullar ◽  
Robert S Chapkin ◽  
...  
Keyword(s):  
16S Rrna ◽  
Sample Size ◽  
Microbial Communities ◽  
State Of The Art ◽  
Metagenomic Data ◽  
Data Sets ◽  
Sequencing Data ◽  
Sample Data

AbstractWe propose a Bayesian method for the classification of 16S rRNA metagenomic profiles of bacterial abundance, by introducing a Poisson-Dirichlet-Multinomial hierarchical model for the sequencing data, constructing a prior distribution from sample data, calculating the posterior distribution in closed form; and deriving an Optimal Bayesian Classifier (OBC). The proposed algorithm is compared to state-of-the-art classification methods for 16S rRNA metagenomic data, including Random Forests and the phylogeny-based Metaphyl algorithm, for varying sample size, classification difficulty, and dimensionality (number of OTUs), using both synthetic and real metagenomic data sets. The results demonstrate that the proposed OBC method, with either noninformative or constructed priors, is competitive or superior to the other methods. In particular, in the case where the ratio of sample size to dimensionality is small, it was observed that the proposed method can vastly outperform the others.Author summaryRecent studies have highlighted the interplay between host genetics, gut microbes, and colorectal tumor initiation/progression. The characterization of microbial communities using metagenomic profiling has therefore received renewed interest. In this paper, we propose a method for classification, i.e., prediction of different outcomes, based on 16S rRNA metagenomic data. The proposed method employs a Bayesian approach, which is suitable for data sets with small ration of number of available instances to the dimensionality. Results using both synthetic and real metagenomic data show that the proposed method can outperform other state-of-the-art metagenomic classification algorithms.

scholarly journals Mumame: a software tool for quantifying gene-specific point-mutations in shotgun metagenomic data

2019 ◽  
Vol 3 ◽  
Author(s):  
Shruthi Magesh ◽  
Viktor Jonsson ◽  
Johan Bengtsson-Palme
Keyword(s):  
Microbial Communities ◽  
Point Mutations ◽  
Software Tool ◽  
Metagenomic Data ◽  
Data Sets ◽  
Resistance Mutations ◽  
Shotgun Metagenomics ◽  
Key Factor ◽  
Detection Of Mutations ◽  
And Function

Metagenomics has emerged as a central technique for studying the structure and function of microbial communities. Often the functional analysis is restricted to classification into broad functional categories. However, important phenotypic differences, such as resistance to antibiotics, are often the result of just one or a few point mutations in otherwise identical sequences. Bioinformatic methods for metagenomic analysis have generally been poor at accounting for this fact, resulting in a somewhat limited picture of important aspects of microbial communities. Here, we address this problem by providing a software tool called Mumame, which can distinguish between wildtype and mutated sequences in shotgun metagenomic data and quantify their relative abundances. We demonstrate the utility of the tool by quantifying antibiotic resistance mutations in several publicly available metagenomic data sets. We also identified that sequencing depth is a key factor to detect rare mutations. Therefore, much larger numbers of sequences may be required for reliable detection of mutations than for most other applications of shotgun metagenomics. Mumame is freely available online (http://microbiology.se/software/mumame).

scholarly journals The impact of sequencing depth on the inferred taxonomic composition and AMR gene content of metagenomic samples

2019 ◽  
Vol 14 (1) ◽  
Author(s):  
H. Soon Gweon ◽  
◽  
Liam P. Shaw ◽  
Jeremy Swann ◽  
Nicola De Maio ◽  
...  
Keyword(s):  
Data Processing ◽  
Open Source ◽  
Open Source Software ◽  
River Sediment ◽  
Taxonomic Composition ◽  
Sequencing Depth ◽  
Gene Content ◽  
Metagenomic Data ◽  
Software Pipeline ◽  
Shotgun Metagenomics

Abstract Background Shotgun metagenomics is increasingly used to characterise microbial communities, particularly for the investigation of antimicrobial resistance (AMR) in different animal and environmental contexts. There are many different approaches for inferring the taxonomic composition and AMR gene content of complex community samples from shotgun metagenomic data, but there has been little work establishing the optimum sequencing depth, data processing and analysis methods for these samples. In this study we used shotgun metagenomics and sequencing of cultured isolates from the same samples to address these issues. We sampled three potential environmental AMR gene reservoirs (pig caeca, river sediment, effluent) and sequenced samples with shotgun metagenomics at high depth (~ 200 million reads per sample). Alongside this, we cultured single-colony isolates of Enterobacteriaceae from the same samples and used hybrid sequencing (short- and long-reads) to create high-quality assemblies for comparison to the metagenomic data. To automate data processing, we developed an open-source software pipeline, ‘ResPipe’. Results Taxonomic profiling was much more stable to sequencing depth than AMR gene content. 1 million reads per sample was sufficient to achieve < 1% dissimilarity to the full taxonomic composition. However, at least 80 million reads per sample were required to recover the full richness of different AMR gene families present in the sample, and additional allelic diversity of AMR genes was still being discovered in effluent at 200 million reads per sample. Normalising the number of reads mapping to AMR genes using gene length and an exogenous spike of Thermus thermophilus DNA substantially changed the estimated gene abundance distributions. While the majority of genomic content from cultured isolates from effluent was recoverable using shotgun metagenomics, this was not the case for pig caeca or river sediment. Conclusions Sequencing depth and profiling method can critically affect the profiling of polymicrobial animal and environmental samples with shotgun metagenomics. Both sequencing of cultured isolates and shotgun metagenomics can recover substantial diversity that is not identified using the other methods. Particular consideration is required when inferring AMR gene content or presence by mapping metagenomic reads to a database. ResPipe, the open-source software pipeline we have developed, is freely available (https://gitlab.com/hsgweon/ResPipe).

scholarly journals Struo: a pipeline for building custom databases for common metagenome profilers

2019 ◽  
Author(s):  
Jacobo de la Cuesta-Zuluaga ◽  
Ruth E. Ley ◽  
Nicholas D. Youngblut
Keyword(s):  
Microbial Diversity ◽  
Microbial Communities ◽  
Source Code ◽  
Functional Information ◽  
Microbial Genomes ◽  
Link Type ◽  
Public Repositories

AbstractSummaryTaxonomic and functional information from microbial communities can be efficiently obtained by metagenome profiling, which requires databases of genes and genomes to which sequence reads are mapped. However, the databases that accompany metagenome profilers are not updated at a pace that matches the increase in available microbial genomes. To address this, we developed Struo, a modular pipeline that automatizes the acquisition of genomes from public repositories and the construction of custom databases for multiple metagenome profilers. The use of custom databases that broadly represent the known microbial diversity by incorporating novel genomes results in a substantial increase in mappability of reads in synthetic and real metagenome datasets.Availability and implementationSource code available for download at https://github.com/leylabmpi/Struo. Custom GTDB databases available at http://ftp.tue.mpg.de/ebio/projects/struo/[email protected]

Business analysis as an effective data processing tool

2020 ◽  
pp. 64-70
Author(s):  
Ponomarenko ◽  
Teleus
Keyword(s):  
Data Processing ◽  
Business Intelligence ◽  
Large Data ◽  
Comprehensive Analysis ◽  
Large Data Sets ◽  
Data Sets ◽  
Business Analytics ◽  
Software Products ◽  
Data Processing Tool ◽  
Analysis System

The subject of the research is the approach to the possibility of using business intelligence for integrated data processing and analysis in order to optimize the company’s activities. The purpose of writing this article is to study the concept of the BI-systems peculiarities use as one of the advanced approaches to the pro- cessing and analysis of large data sets that are continuously accumulated from various sources. Methodology. The research methodology is system-structural and comparative analyzes (to study the application of BI-systems in the process of working with large data sets); monograph (the study of various software solutions in the market of business intelligence); economic analysis (when assessing the pos- sibility of using business intelligence systems to strengthen the competitive position of companies). The scientific novelty consists the features of using the business analytics model in modern conditions to optimize the activities of companies through the use of complex information, which in many cases is unstructured, are identified. The main directions of working with big data are disclosed, starting from the stage of collection and storage in specialized repositories, and ending with a comprehensive analysis of information. The main advantages of using dashboards in the process of demonstrating research results are given. A comprehensive analysis of software products in the business intelligence market has been carried out. Conclusions. The use of business intelligence allows companies to optimize their activities by making effective management decisions. The availability of a large number of BI tools al- lows company to adapt the analysis system in accordance with available data and existing needs of the company. Software solutions make it possible to build dash- boards with the settings of the selected system of indicators.

An accurate and fast alignment-free method for profiling microbial communities

2017 ◽  
Vol 15 (03) ◽  
pp. 1740001 ◽  
Author(s):  
Diem-Trang Pham ◽  
Shanshan Gao ◽  
Vinhthuy Phan
Keyword(s):  
Microbial Communities ◽  
Accurate Method ◽  
Metagenomic Data ◽  
Quadratic Programs ◽  
Microbial Genomes ◽  
Microbial Species ◽  
Alignment Free ◽  
Human Dna ◽  
Computationally Expensive ◽  
Reference Genomes

Determining abundances of microbial genomes in metagenomic samples is an important problem in analyzing metagenomic data. Although homology-based methods are popular, they have shown to be computationally expensive due to the alignment of tens of millions of reads from metagenomic samples to reference genomes of hundreds to thousands of environmental microbial species. We introduce an efficient alignment-free approach to estimate abundances of microbial genomes in metagenomic samples. The approach is based on solving linear and quadratic programs, which are represented by genome-specific markers (GSM). We compared our method against popular alignment-free and homology-based methods. Without contamination, our method was more accurate than other alignment-free methods while being much faster than a homology-based method. In more realistic settings where samples were contaminated with human DNA, our method was the most accurate method in predicting abundance at varying levels of contamination. We achieve higher accuracy than both alignment-free and homology-based methods.

scholarly journals Soil Microbial Community Responses to a Decade of Warming as Revealed by Comparative Metagenomics

2013 ◽  
Vol 80 (5) ◽  
pp. 1777-1786 ◽  
Author(s):  
Chengwei Luo ◽  
Luis M. Rodriguez-R ◽  
Eric R. Johnston ◽  
Liyou Wu ◽  
Lei Cheng ◽  
...  
Keyword(s):  
Climate Change ◽  
Microbial Communities ◽  
Molecular Mechanisms ◽  
Cellulose Degradation ◽  
Global Climate ◽  
Soil Microbial Communities ◽  
Predictive Ability ◽  
Metagenomic Data ◽  
Data Sets ◽  
Soil Microbial

ABSTRACTSoil microbial communities are extremely complex, being composed of thousands of low-abundance species (<0.1% of total). How such complex communities respond to natural or human-induced fluctuations, including major perturbations such as global climate change, remains poorly understood, severely limiting our predictive ability for soil ecosystem functioning and resilience. In this study, we compared 12 whole-community shotgun metagenomic data sets from a grassland soil in the Midwestern United States, half representing soil that had undergone infrared warming by 2°C for 10 years, which simulated the effects of climate change, and the other half representing the adjacent soil that received no warming and thus, served as controls. Our analyses revealed that the heated communities showed significant shifts in composition and predicted metabolism, and these shifts were community wide as opposed to being attributable to a few taxa. Key metabolic pathways related to carbon turnover, such as cellulose degradation (∼13%) and CO2production (∼10%), and to nitrogen cycling, including denitrification (∼12%), were enriched under warming, which was consistent with independent physicochemical measurements. These community shifts were interlinked, in part, with higher primary productivity of the aboveground plant communities stimulated by warming, revealing that most of the additional, plant-derived soil carbon was likely respired by microbial activity. Warming also enriched for a higher abundance of sporulation genes and genomes with higher G+C content. Collectively, our results indicate that microbial communities of temperate grassland soils play important roles in mediating feedback responses to climate change and advance the understanding of the molecular mechanisms of community adaptation to environmental perturbations.

Sign in / Sign up

Export Citation Format

Share Document