Block Storage Optimization and Parallel Data Processing and Analysis of Product Big Data Based on the Hadoop Platform

The traditional distributed database storage architecture has the problems of low efficiency and storage capacity in managing data resources of seafood products. We reviewed various storage and retrieval technologies for the big data resources. A block storage layout optimization method based on the Hadoop platform and a parallel data processing and analysis method based on the MapReduce model are proposed. A multireplica consistent hashing algorithm based on data correlation and spatial and temporal properties is used in the parallel data processing and analysis method. The data distribution strategy and block size adjustment are studied based on the Hadoop platform. A multidata source parallel join query algorithm and a multi-channel data fusion feature extraction algorithm based on data-optimized storage are designed for the big data resources of seafood products according to the MapReduce parallel frame work. Practical verification shows that the storage optimization and data-retrieval methods provide supports for constructing a big data resource-management platform for seafood products and realize efficient organization and management of the big data resources of seafood products. The execution time of multidata source parallel retrieval is only 32% of the time of the standard Hadoop scheme, and the execution time of the multichannel data fusion feature extraction algorithm is only 35% of the time of the standard Hadoop scheme.

GenoVault: a cloud based genomics repository

GenoVault is a cloud-based repository for handling Next Generation Sequencing (NGS) data. It is developed using OpenStack-based private cloud with various services like keystone for authentication, cinder for block storage, neutron for networking and nova for managing compute instances for the Cloud. GenoVault uses object-based storage, which enables data to be stored as objects instead of files or blocks for faster retrieval from different distributed object nodes. Along with a web-based interface, a JavaFX-based desktop client has also been developed to meet the requirements of large file uploads that are usually seen in NGS datasets. Users can store files in their respective object-based storage areas and the metadata provided by the user during file uploads is used for querying the database. GenoVault repository is designed taking into account future needs and hence can scale both vertically and horizontally using OpenStack-based cloud features. Users have an option to make the data shareable to the public or restrict the access as private. Data security is ensured as every container is a separate entity in object-based storage architecture which is also supported by Secure File Transfer Protocol (SFTP) for data upload and download. The data is uploaded by the user in individual containers that include raw read files (fastq), processed alignment files (bam, sam, bed) and the output of variation detection (vcf). GenoVault architecture allows verification of the data in terms of integrity and authentication before making it available to collaborators as per the user’s permissions. GenoVault is useful for maintaining the organization-wide NGS data generated in various labs which is not yet published and submitted to public repositories like NCBI. GenoVault also provides support to share NGS data among the collaborating institutions. GenoVault can thus manage vast volumes of NGS data on any OpenStack-based private cloud.

A Survey on Different Storage Architectures

This paper surveys different types of data storage architectures used in today’s world to store data of various types and presents an in-depth analysis of the use cases and drawbacks of the various storage architectures in different scenarios. We also examine the limitations of the traditional storage architectures in today’s world and also discuss the modern solutions to these problems. In this survey paper, we have provided a detailed comparison of three different storage architectures namely file storage, block storage, and object storage. This paper provides sufficient information to the reader to be able to choose among these architectures for storing their data as per the use case.

An Efficient Data Analysis Framework for Online Security Processing

Industrial cloud security and internet of things security represent the most important research directions of cyberspace security. Most existing studies on traditional cloud data security analysis were focused on inspecting techniques for block storage data in the cloud. None of them consider the problem that multidimension online temp data analysis in the cloud may appear as continuous and rapid streams, and the scalable analysis rules are continuous online rules generated by deep learning models. To address this problem, in this paper we propose a new LCN-Index data security analysis framework for large scalable rules in the industrial cloud. LCN-Index uses the MapReduce computing paradigm to deploy large scale online data analysis rules: in the mapping stage, it divides each attribute into a batch of analysis predicate sets which are then deployed onto a mapping node using interval predicate index. In the reducing stage, it merges results from the mapping nodes using multiattribute hash index. By doing so, a stream tuple can be efficiently evaluated by going over the LCN-Index framework. Experiments demonstrate the utility of the proposed method.

GenoVault: A Cloud Based Genomics Repository

GenoVault is a cloud-based repository for handling Next Generation Sequencing (NGS) data. It is developed using OpenStack based private cloud with various services like keystone for authentication, cinder for block storage, neutron for networking and nova for managing compute instances for the Cloud. GenoVault uses object-based storage, which enables data to be stored as objects instead of files or blocks for faster retrieval from different distributed object nodes. Along with a web-based interface JavaFX-based desktop client has also been developed to meet the requirements of large file uploads ( > 5 GB) that is usually seen in NGS datasets. Users can store as many as one million files in their respective object based storage areas and the metadata provided by the user during file uploads is used for querying the database. GenoVault repository is designed taking into account future needs and hence can scale both vertically and horizontally without any need for modification in the design. Users have an option to make the data shareable to the public or restrict the access as private. Data security is ensured as every container is a separate entity in object-based storage architecture also supported by secured file transfer protocol during data upload and download. The data is uploaded by the user in their individual containers that include raw read files (fastq), processed alignment files (bam, sam, bed) and output of variation detection (vcf). GenoVault architecture allows verification of the data in terms of integrity and authentication before making it available to collaborators as per user permissions. GenoVault is useful for maintaining the organization wide NGS data generated by experiments in various labs which is not yet published and submitted to public repositories like NCBI. GenoVault also provides support to share NGS data among the collaborating institutions. GenoVault can thus manage vast volumes of NGS data on any OpenStack-based private cloud.